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1.  Introduction 


Effects-Based  Operations  (EBO)  provides  a  new  paradigm  for  conceiving  and  conducting 
military  campaigns.  Supplement  One  to  the  Commander’s  Handbook  for  an  Effects  Based 
Approach  to  Joint  Operations  (13  March  2006)  defines  EBO  as,  “Operations  that  are  planned, 
executed,  assessed,  and  adapted  based  on  a  holistic  understanding  of  the  operational  environment 
in  order  to  influence  or  change  system  behavior  or  capabilities  using  the  integrated  application  of 
select  instruments  of  power  to  achieve  directed  policy  aims.”  The  handbook  also  points  out  that 
EBO  is  comprised  of  a  continuous  loop  of  three  basic  processes:  planning,  execution  and 
assessment.  The  focus  of  this  effort  is  on  the  final  leg  of  the  EBO  triangle,  assessment. 

Within  the  air  component  of  the  joint  forces  structure,  the  Strategy  Plans  Team  (SPT)  of  the 
Strategy  Division  has  the  responsibility  for  translating  the  Joint  Forces  Commander’s  (JFC) 
desired  effects  into  a  strategy  of  operational  objectives,  tactical  objectives,  and  tactical  tasks.  Air 
strategy  objectives  and  tasks  are  developed  through  a  Course  of  Action  (CO  A)  analysis  where 
alternative  approaches  are  specified  and  where  the  strengths  and  weaknesses  of  each  are 
compared  to  determine  which  is  superior. 

EBO  is  a  performance-based  process  with  measurable  goals.  The  SPT  specifies  Measures  of 
Effectiveness  (MoE)  to  assess  progress  toward  goals.  The  Operational  Assessment  Team  (OAT) 
develops  and  implements  a  plan  that  gathers  data  on  indicators  and  other  factors  defined  by 
planners  to  generate  the  performance  metrics.  Thus,  the  OAT  implements  the  feedback  loop  that 
allows  the  SPT  to  evaluate  how  their  plan  is  progressing  and  make  adjustments  where  necessary. 
Current  strategy  planning  and  assessment  processes  are  mostly  manual.  This  is  due,  in  part,  to 
the  complex,  somewhat  fuzzy  nature  of  the  strategy  processes.  An  important  role  of  the 
Operational  Effect  Assessment  Visualization  Tool  (OEAVT)  envisioned  by  the  original  Program 
Research  and  Development  Announcements  (PRD A)  was  to  help  formalize  the  processes,  data, 
and  assessments  used  by  the  Strategy  Division  by  providing  a  top  level  visualization  of  the  battle 
space. 

While  there  are  a  variety  of  problems  the  strategy  visualization  tools  must  solve,  a  problem 
common  to  both  the  SPT  and  OAT  is  that  they  currently  are  “disconnected”  from  the  Air  and 
Space  Operations  Center  (AOC)  data  environment  (e.g.,  the  Theater  Battle  Management  Core 
System  and  other  key  systems).  By  disconnected,  we  mean  that  strategy  planning  products, 
which  contain  the  MoEs,  are  not  entered  into  the  AOC  data  environment.  Also,  by  having  no 
visibility  into  the  AOC  data  environment,  the  OAT  is  prevented  from  establishing  automated 
collection  of  the  air  campaign  results  required  to  make  assessments. 

One  result  of  this  lack  of  access  is  an  inability  to  link  a  given  Air  Tasking  Order  (ATO)  to  the 
broader  effects-based  plan.  This  loss  of  operational  context  results  in  “open-loop”  air 
campaigns,  in  which  plans  cannot  be  adapted  to  the  evolution  of  operations.  Thus,  AOC  Combat 
Operations  Division  personnel  have  access  to  the  ATO  but  they  do  not  have  access  to 
information  about  a  particular  target’s  importance  to  the  broader  campaign  strategy  desired 
effects  or  to  the  status  of  that  target  with  respect  to  the  goals  of  the  JFACC  and  JFC.  Thus,  it  is 
possible  to  adversely  affect  implementation  of  the  plan  by  re-tasking  strike  assets  to  lower  valued 
targets. 
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A  second  significant  impact  is  that  it  is  difficult  for  the  OAT  to  assess  how  well  effects  are  being 
achieved  because  (1)  they  lack  insight  into  how  effects  are  implemented  through  a  given  ATO 
and  (2)  the  AOC  data  environment  does  not  have  the  capability  to  record  and  organize  execution 
data,  strike  results  and  non-kinetic  activities  in  a  form  readily  interpretable  by  the  assessment 
team.  Essentially,  this  means  that  the  feedback  loop,  which  makes  EBO  possible,  is  broken  in 
the  AOC. 
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2.  Purpose  of  the  OEAVT  Program 


The  goal  of  the  OEAVT  program  was  to  ereate  innovative,  decision-aetionable  work  support 
systems  that  would  enable  the  AOC  OAT  to  plan  and  earry  out  assessment  of  effeets-based  plans 
within  a  dynamieally  evolving  air  campaign.  This  would  be  accomplished  by  providing 
intuitive,  high-level  visualizations  of  mission  effects,  mechanisms  and  their  interrelationships 
during  the  air  operations  planning  and  assessment  phases  of  successive  ATO  cycles. 

SAlC’s  technical  approach  to  developing  OEAVT  was  guided  by  a  desire  to  achieve  three  major 
objectives.  These  were  to  (1)  thoroughly  understand  the  requisite  OA  knowledge,  skills,  and 
experience  that  would  enable  design  and  development  of  an  innovative,  effective  OA  support 
system;  (2)  apply  proven,  reliable  engineering  processes  that  would  ensure  effective 
development  of  the  OEAVT;  and  (3)  implement  an  evaluation  methodology  that  would  expose 
OEAVT  to  a  Test  Readiness  Level  6  environment,  so  as  to  demonstrate  its  operational 
effectiveness  and  create  opportunities  for  transition  to  the  operational  setting.  The  baseline  plan 
for  achieving  these  three  objectives  consisted  of  five  technical  tasks: 

2.1  Task  1 :  Translate  Cognitive  Task  Anaiysis  (CTA)  into  System  Engineering 
Requirements. 

2.1.1  Conduct  Cognitive  Systems  Analysis  (CSA)  of  Operational  Assessment  teams. 


Task  1  focused  on  gathering  operational,  behavioral  and  work  information  needed  to  understand 
the  Operational  Assessment  domain,  the  tasks  involved  in  carrying  out  assessment  and  the 
opportunities,  strategies  and  constraints  involved  in  planning  for  and  implementing  an 
assessment  program  for  particular  air  campaigns.  The  CSA  included  analyses  of  OA  work 
domains;  analyses  of  assessor  tasks  and  strategies;  and  analyses  of  the  socio-technical  and 
organizational  factors  important  in  the  conduct  of  operational  assessment.  An  additional  goal  of 
the  effort  was  identification  of  critical  measures  of  performance  (i.e.,  key  performance  factors) 
that  could  be  used  as  metrics  for  OAT  tools  and  operational  effectiveness. 


2.1.2  Validate  CSA  results. 


The  CSA  results  obtained  were  validated  prior  to  the  initiation  of  system  development. 
Validation  was  accomplished  by  subjecting  the  results  to  review  by  Air  Force  uniformed  and/or 
civilian  personnel  who  were  Subject  Matter  Experts  (SME)  in  Strategy  Division  operations. 
Results  obtained  in  the  validation  process  were  used  to  update  and/or  correct  results  of  the 
analysis  conducted. 
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2.1.3  Compile  CSA  Results  into  a  System  Engineering  Tool/Database 


This  task  was  to  ensure  that  CSA  results  were  included  in  OEAVT  system  requirements.  This 
was  required  for  effective  development  of  visualization  support  for  the  OA  domain.  We  relied 
on  both  our  own  process  and  complimentary  system  engineering  tools  to  accomplish  this 
objective.  Our  review  of  available  system  and  software  engineering  tools  revealed  that  the 
CORE® '  tool  was  the  most  suitable  for  incorporating  and  transforming  the  results  of  a  CSA  into 
system  engineering  content  suitable  for  requirements  development. 

2.2  Task  2:  Evaluate  OEAVT  Candidate  Solutions 


The  requirement  for  Task  2  as  laid  out  in  the  Statement  of  Work,  (SOW),  states  that  SAIC  is  to 
use  the  system  engineering  requirements  derived  from  the  CTA  developed  in  Task  1,  to 
accomplish  an  analysis  to  determine  solution  categories.  Based  on  these  solution  categories, 
SAIC  is  to  develop  and  create  prototype  display  and  database  design  concepts  and  analytically 
evaluate  each  to  determine  viability. 

Our  approach  to  evaluating  candidate  solution  categories  is  an  explicit  merger  of  cognitive, 
human  factors,  and  system  and  software  engineering  processes.  This  merger  ensures  that 
insights  into  OEAVT  user  needs  derived  from  the  CTA  are  converted  into  sound  visualization 
and  other  concepts.  One  of  the  main  systems  of  records  at  the  time  was  Information  Warfare 
Planning  Capability  (IWPC).  This  system  was  investigated  as  a  starting  point.  Review  of  the 
previous  work  done  by  ManTech  was  also  accomplished.  It  was  decided  that  a  significant 
conceptual  and  technological  gap  existed  in  the  current  AOC  tools  from  what  the  output  of  the 
CTA  indicated  was  needed.  We  decided  to  design  our  own  set  of  User  interfaces  extensive 
prototyping  prior  to  development.  These  prototypes  were  manipulated  and  critiqued  by 
prospective  users  and  our  own  OA  SMEs. 

2.3  Task  3:  Develop  OEAVT  Solutions 


This  task  involved  developing  the  OEAVT  software  system  components.  Capability  Maturity 
Model®  Integration  (CMMI®”^)  Level  3  processes  were  applied  to  development  of  the  system. 
Software  requirements  were  generated  from  the  System  Requirements  and  a  software 
architectural  design  then  followed.  The  software  components  were  developed  using  Java^'^^  and 
the  individual  units  were  tested  to  verify  functionality.  This  task  also  included  the  integration  of 
software  components  that  make  up  the  OEAVT  system.  See  Section  12  for  more  detail  on  the 
development  of  the  OEAVT  System. 


1  CORE  is  a  registered  trademark  of  Vitech  Corporation,  2070  Chain  Bridge  Road  Suite  100  Vienna,  Virginia 
22182 

2  Java  is  a  trademark  of  Sun  Microsystems  Inc. 
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2.4  Task  4:  Conduct  Operational  Test 


The  SOW  requirement  for  this  Task  was  for  the  OEAVT  system  software  to  operate  suceessfully 
at  an  operational  test.  The  SOW  states,  that  the  OEAVT  software  system,  eompleted  as  part  of 
the  spiral  development  process,  shall  be  integrated  with  the  Task  3  database  and  evaluated  within 
an  operational  context  using  a  scenario-specific  database.  Successful  completion  of  the 
operational  test  shall  result  in  a  Technology  Readiness  Level  (TRL)  6  system.  At  TRL  6,  a 
representative  model  or  prototype  system  would  be  tested  in  a  relevant  environment.  The 
demonstration  might  represent  an  actual  system  application,  or  it  might  only  be  similar  to  the 
planned  application,  but  use  the  same  technologies. 

To  satisfy  this  SOW  requirement,  SAIC  conducted  the  operational  testing  at  the  Warfighter 
Analysis  of  Innovative  Technologies  and  Concepts  (WAIT  &  C)  event  at  Langley,  AFB. 
Members  of  the  SAIC  team  attended  this  event  as  part  of  an  ongoing  series  of  technology 
demonstrations  carried  out  by  the  GCIC.  Focus  of  this  event  was  on  operational  planning  and 
assessment.  Eight  SMEs  were  invited  to  the  event  to  serve  as  evaluators  of  the  technologies 
under  demonstration.  The  OEAVT  system  was  successfully  demonstrated  for  each  SME. 
Opportunities  for  improvement  were  recorded  and  then  prioritized  and  weighted  against  our 
current  solution  and  available  funding.  A  decision  was  then  made  as  to  whether  the 
improvement  could  be  incorporated  into  the  OEAVT  development.  Operational  SMEs  provided 
an  excellent  opportunity  to  showcase  OEAVT’ s  new  Operational  Assessment  capabilities. 

2.5  Task  5:  Report  Results 


The  output  of  Task  5  is  a  final  technical  report  documenting  results  of  the  OEAVT  system 
development  effort,  as  well  as  the  engineering  methods  used  to  produce  the  system.  This  final 
report  also  includes  a  copy  of  the  data  used  to  document  results  in  a  format  readily  usable  by 
future  Operational  Assessment  team’s  visualization  tool  developers. 
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3.  Document  Overview 


In  the  remainder  of  this  document  we  report  on  the  development  of  OEAVT  and  present  the 
system  that  has  resulted  from  that  development  effort.  We  begin  with  a  discussion  of  the 
differences  between  Tactical  Assessment  (TA)  and  Operational  Assessment  (OA)  in  Section  5. 
This  sets  the  context  for  the  analyses,  engineering  and  development  discussions  that  follow.  This 
description  of  differences  is  followed  by  a  discussion  of  the  OEAVT  team  in  Section  6,  including 
members  from  the  prime  contractor  and  each  sub-contractor.  This  will  allow  us  to  describe  the 
roles  played  by  each  team  member  in  the  development  of  the  system. 

We  then  follow  in  Section  7  with  a  brief  discussion  of  some  of  the  challenges  that  we 
encountered  during  the  evolution  of  OEAVT.  This  was  a  complex  program,  as  open  systems 
typically  are,  with  many  “moving  parts.” 

Section  8  outlines  the  general  manner  in  which  the  OEAVT  team  developed  the  system. 
Development  was  based  on  an  integrated  joint  cognitive  system  methodology  that  has  been  used 
successfully  in  a  number  of  system  engineering  efforts  by  members  of  the  SAIC  team.  Section  9 
of  the  report  describes  the  cognitive  and  work  analyses  carried  out  in  the  initial  stages  of 
OEAVT  development. 

A  detailed  description  of  OEAVT  system  engineering  follows  in  Section  10.  The  software 
architecture  of  the  system  is  presented  in  Section  1 1 .  The  final  discussion  of  the  report.  Section 
12,  includes  descriptions  of  the  system  components  that  resulted  from  the  OEAVT  engineering 
and  development  processes.  These  include  the  Theater  Battle  Operations  Net-centric 
Environment-Information  Warfare  Planning  Capability  (TBONE-IWPC)  Indicator  Interface 
(TI3)  module,  the  Action-Effects  Contrast  Visualization  (AECV)  module,  the  Geo-spatial  Effects 
Visualization  (GEV)  module  and  the  Global  Effects  Matrix-Synchronization  (GEM-S)  module. 
The  report  concludes  with  Appendices  containing  detailed  descriptions  of  the  analysis  and 
engineering  products  of  the  program. 
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4.  Operational  Differences  Between  Tactical  Assessment  and 
Operational  Assessment 

The  distinction  between  combat  or  tactical  assessment  (TA)  and  Operational  Assessment  (OA)  is 
often  blurred,  even  by  practicing  USAF  assessors.  The  cognitive  and  behavior  aspects  of  TA 
and  OA  were  both  examined  over  the  course  of  this  effort  to  better  understand  and  define  these 
aspects  of  assessment. 

Operational  assessment  is  the  highest  level  of  assessment  at  the  Joint  Forces  Air  Component 
Commander  (JFACC)  level.  It  provides  plan  status  and  recommendations  to  both  JFACC  and 
Combined  Forces  Commander  (CFC).  Personnel  include  operations  research  analysts,  pilots, 
navigators,  information  operators,  intelligence  analysts,  etc.  They  are  assigned  to  the 
Operational  Assessment  Team  (OAT)  in  the  Strategy  Division.  Their  assessments  derive  from  a 
combination  of  intelligence,  operations,  and  logistics  sources.  According  to  the  Air  Force  EBO 
Lexicon,  OA  is  the  "'Joint  force  components  ’  evaluation  of  the  achievement  of  their  objectives, 
both  tactical  and  operational,  through  assessment  of  effects,  operational  execution, 
environmental  influences,  and  attainment  of  the  objectives  ’  success  indicators,  in  order  to 
develop  strategy  recommendations.  ” 

In  particular,  OA  focuses  on  higher-order  elements  of  a  plan,  defined  in  terms  of  logical 
sequences  of  causes  and  effects.  Based  on  Air  Force  Tactics,  Techniques,  and  Procedures 
(AFTTP)  Air  and  Space  Strategy,  an  operational  line  of  effect  (OLE)  is  defined  as  “a  logical  line 
that  defines,  in  sequence  and  purpose,  the  orientation  of  operational-level  actions  and  associated 
causal  links  and  effects,  tactical  lines  of  effect  and  their  associated  tactical  objectives  and  any 
additional  causal  links  in  the  delivery  of  an  operational  objective.”  This  relationship  is  shown  in 
Figure  1. 


Enabiing 

Tactical  Objective 


Contributory 
Tactical  Objective 


Figure  1 :  The  Targets  and  Assessment  Teams  (TAT)  Collects,  Fuses, 
Synthesizes,  Reports,  and  Formulates  Recommendations  on  Tactical  Tasks  (TT) 
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OA  relies  on  Tactical  Assessment  to  gauge  progress  toward  achieving  Operational  Objectives 
(00s).  It  measures  the  status  of  Tactical  Tasks  and  Tactical  Objectives  (TOs)  to  arrive  at  the 
status  of  00s.  Operational  assessors  must  always  be  mindful,  however,  of  the  reason  for  their 
assessment  of  00s.  Their  fused  results  are  intended  to  be  applied  to  the  status  of  campaign 
effects,  including  end-state  conditions.  The  latter  should  be  the  central  focus  in  all  cases,  to 
determine  the  overall  campaign  status,  how  we  are  supporting  it,  and  when  we  can  decrease  and 
halt  operations. 

TA  is  the  lowest  level  of  assessment  at  the  Joint  Forces  Air  Component  Commander  (JFACC) 
level.  It  provides  target/plan  status  and  recommendations  to  the  OA  Team.  TA  occurs  in  the 
Tactical  Assessment  Cell  (TAC),  which  is  part  of  the  Targets  and  Assessment  Team  (TAT)  in 
the  ISR  Division.  Personnel  primarily  consist  of  intelligence  analysts  and  targeteers.  Their 
information  is  almost  exclusively  derived  from  intelligence  sources.  Based  on  the  Air  Force 
EBO  Lexicon,  TA  is  the  “overall  determination  of  the  success  of  tactical  operations.  ” 

TA  focuses  on  actions  carried  out  in  support  of  tactical  tasks,  and  direct  and  second-order  effects 
of  those  actions  on  planned  targets.  Based  on  Air  Force  Tactics,  Techniques,  and  Procedures 
(AFTTP)  Air  and  Space  Strategy,  a  Tactical  Line  of  Effect  (TLE)  is  the  element  of  the  Effects- 
Based  Operational  Design  (EBOD)  construct  that  captures  and  details  the  operational  design  for 
the  delivery  of  a  TO  along  a  particular  OLE.  A  TLE  is  defined  as  “a  logical  line  that  defines,  in 
sequence  and  purpose,  the  orientation  of  tactical  task  lines  of  effect,  their  associated  intended 
tactical  task  objectives  (TTOs),  and  any  additional  causal  links  in  the  delivery  of  a  tactical 
objective." 

In  the  same  manner  that  the  TLE  is  the  scheme  of  effects  (i.e.,  TTOs)  that  will  deliver  a  TO,  a 
Tactical  Task  Line  of  Effect  (TTLE)  is  the  “logical  line  that  defines,  in  sequence  and  purpose, 
the  orientation  of  tactical  actions,  consequential  direct  and  indirect  effects  and  associated  causal 
linkages  in  the  delivery  of  an  intended  tactical  task  objective.”  TTLEs  will  normally  employ  one 
of  three  basic  effects-chain  constructs  that  will  detail  the  plan  for  delivering  an  intended  discrete 
TTO.  This  sequence  is  shown  in  Figure  2. 


TTLE:  Taciicai  Task  Lina  of  Effect 
TTO-  Tactical  Task  Objective 
DE-  Direct  Effect 
IE-  indirect  Effect 
CL-  Causal  Link 

Figure  2:  Effects-chain  Constructs  Employed  by  TTLE  to  Deiiver  Discrete  TTOs 
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The  boundary  layer  between  TA  and  OA  also  is  worth  noting.  On  the  one  hand,  TA  is  taken  as 
all  the  plan  elements  and  attributes  up  to,  but  excluding  the  Tactical  Objective  (TO),  while  on  the 
other,  it  may  include  the  TO  level  as  well.  Regardless  of  the  delineation,  specific  guidance  must 
be  given  to  all  assessors  involved,  with  consideration  given  for  resource  availability,  capability, 
and  product  delivery. 

TA  feeds  OA  by  means  of  “rolling  up”  tactical-action,  as  well  as  effect  status.  Tactical  assessors 
gauge  the  progress  of  Tactical  Tasks  (TTs)  by  monitoring  changes  in  target  status.  This  is  done 
through  collection,  fusion,  and  synthesis  of  tactical  evidence.  Reporting,  accompanied  by 
recommendations,  is  then  made  to  the  OAT.  However,  this  is  by  no  means  the  sole  basis  of  OA. 
Rather  independent  evidence  must  be  analyzed  at  each  level,  with  assessments  based  on  a  broad 
spectrum  of  inputs. 

4.1  Behavioral  and  Cognitive  Differences  Between  TA  and  OA 


In  addition  to  the  operational  differences  between  tactical  and  Operational  Assessment  discussed 
above,  there  are  several  cognitive  and  behavioral  differences  as  well.  These  differences  fall  into 
five  categories:  Purpose  of  analysis;  objects  of  analysis;  workflow  characteristics;  types  of 
decisions  made  in  each  type  of  assessment  and  the  processes  used  to  reach  decisions;  and  the 
criteria  for  success. 

4.1.1  Purpose  of  analysis 


Each  type  of  assessment  differs  in  purpose.  Tactical  assessment  (TA)  is  focused  primarily  on 
estimating  the  physical,  functional  and  mission-related  capabilities  of  individual  target  systems 
in  response  to  attacks  carried  out  against  those  systems.  Operational  assessment  focuses  on 
determining  whether  progress  toward  planned,  desired  effects  is  proceeding  satisfactorily  and,  if 
not,  what  are  the  sources  of  the  shortfalls  and  what  changes  to  the  operational  plan  can  be  made 
to  achieve  the  desired  effects  in  a  timely  manner.  The  emphasis  on  destroying,  disrupting  or 
neutralizing  individual  systems  through  the  application  of  air  power  often  will  focus  the  attention 
and  efforts  of  the  TA  team  on  kinetic  effects.  Any  focus  on  non-kinetic  effects  often  is  implicit. 
The  focus  of  operational  assessment,  on  the  other  hand,  is  explicitly  both  kinetic  and  non-kinetic 
in  its  pursuit  of  effects.  From  the  point  of  view  of  TA,  an  abandoned  radar  should  be  destroyed. 
From  the  OA  point  of  view,  a  radar  that  is  not  being  used  to  threaten  friendly  air  assets 
represents  a  desirable  outcome  and  progress  toward  an  effect.  That  the  radar  is  abandoned 
because  leaflets  convinced  the  crew  to  return  home,  rather  than  risk  their  lives  to  defend  an 
assigned  area,  makes  little  difference  to  the  achievement  of  the  desired  effect.  Non-kinetic 
means  enjoy  the  same  effectiveness  status  as  kinetic  means  for  the  operational  assessor.  The 
emphasis  of  TA  on  single  systems,  or  small  collections  of  systems,  indicates  a  focus  on  local 
objectives  and  considerations.  The  operational  assessor,  on  the  other  hand,  focuses  on  more 
global  considerations  such  as  systems  of  systems  (SoS),  socio-economic  structures  and  political 
considerations.  There  also  is  a  concern  with  interactions  between  these  “systems”  on  the  part  of 
the  operational  assessor.  Tactical  assessment  focuses  on  first-order  effects.  Operational 
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assessment  attempts  to  account  for  effects  at  the  second-order  and  beyond.  Thus,  complex 
causal  analysis  is  more  of  a  concern  for  OA  than  for  TA.  A  major  purpose  driving  TA  is  a 
concern  with  reconstitution,  re-assignment  and  re-attack.  That  is;  when  will  a  system,  previously 
disabled,  be  re-enabled;  when  should  a  system  be  attacked  again;  and  what  assets  should  be 
assigned  or  re-assigned  to  carry  out  a  strike.  Conversely,  a  major  concern  of  OA  is  with  re¬ 
allocation.  The  fundamental  question  is:  In  order  to  maintain  (or  regain)  satisfactory  progress 
toward  desired  effects,  what  and  how  do  resources  need  to  be  re-allocated,  relative  to  current 
allocations. 

4. 1 .2  Objects  of  analysis 


These  two  types  of  assessment  also  differ  in  the  objects  of  their  analysis.  The  primary  objects  of 
analysis  in  TA  are  physical  systems:  radars,  airfields,  command  buildings,  power  stations  and 
other  physical  installations.  Operational  assessment,  by  contrast,  takes  both  physical  and 
conceptual  systems  as  its  objects  of  analysis.  In  addition  to  the  physical  objects  enumerated 
above,  OA  also  concerns  itself  with  political,  economic  and  social  systems.  The  systems  that 
each  type  of  assessment  considers  will  range  from  single  systems,  in  the  case  of  TA,  to  systems- 
of-systems  (SoS),  in  the  case  of  OA.  The  operational  assessor  also  will  be  concerned  with  the 
interactions  among  systems  as  well  as  the  systems  themselves.  When  considering  networks,  the 
tactical  assessor  will  concentrate  on  single  nodes  of  networks.  Operational  assessors, 
conversely,  will  be  concerned  with  entire  networks  and  the  relationships  between  these  networks. 

4. 1 .3  Workflow  characteristics 


These  two  types  of  assessment  also  differ  in  their  workflow  characteristics.  The  workflow  that 
best  characterizes  TA  is  linear  in  nature,  proceeding  from  early  analysis  Bomb  Damage 
Assessment  (BDA)  through  successive  stages  to  late  analysis  Mission  Effects  Assessment 
(MEA).  The  workflow  of  OA  is  more  non-linear  in  character.  The  operational  assessor 
performs  analysis  on  plan  elements  that  are  not  arranged  in  a  strict  hierarchy  (although  there  is 
often  some  priority  set  on  operational  objectives).  Thus,  analytical  focus  will  move  among  plan 
elements  and  other  tasks  in  a  non-linear  manner.  This  implies  that  TA  will  be  subject  to  a 
stricter  sequencing  than  will  be  the  case  for  OA.  For  example,  in  TA  one  performs  BDA  before 
Functional  Damage  Assessment  (FDA).  Each  stage  follows,  more  or  less,  logically  on  another. 
Operational  assessors  can  better  be  described  as  exhibiting  a  “random  access”  of  analytical 
objects  as  information  about  these  objects  becomes  available  or  as  other  requirements  prompt 
analysis.  The  final  difference,  again  related  to  the  discussion  above,  is  that  information  arrival 
tends  to  follow  a  synchronous  pattern  in  TA.  Early  information  bears  on  the  process  of  BDA, 
later  information  is  relevant  to  other  processes  (Physical  Damage  Assessment,  FDA,  MEA,  etc.). 
Information  availability  is  more  asynchronous  in  OA.  Information  relevant  to  the  assessment  of 
operational  plan  elements  is  made  available  to  the  OAT  in  a  first-arrival,  first-announced 
manner,  more  or  less  independent  of  plan  element  priority,  OAT  request,  the  ATO  in  force,  point 
in  the  air  campaign  or  other  factors. 
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4.1.4  Types  of  decisions  and  decision  processes 


These  differences  in  purpose,  objects  of  analysis  and  workflow  naturally  lead  to  differences  in 
the  types  of  decisions  between  TA  and  OA.  The  decisions  made  by  tactical  assessors  are 
concerned  with  sequencing  and/or  scheduling,  state  estimation,  and  assignment.  These  derive 
from  the  emphases  in  TA  on  reconstitution,  re-assignment  and  re-attack.  These  requirements 
drive  tactical  assessors  to  be  concerned  with  the  problem  of  how  to  manage  assets  in  ways  that 
will  satisfy  these  emphases,  while  maintaining  the  integrity  of  the  effects-based  plan  and 
operational  tempo  of  the  overall  air  campaign.  The  major  criterion  underlying  these  types  of 
decisions  for  the  tactical  assessor,  then,  will  be  that  of  optimization.  The  decision  categories  that 
apply  to  OA,  on  the  other  hand,  include  decisions  focusing  on  plan  modifications,  diagnosis  and 
troubleshooting,  and  allocation.  In  evaluating  an  Effects-Based  Plan  (EBP)  the  OAT  will 
inevitably  encounter  deviations  of  actual  results  from  those  planned  to  occur,  with  respect  to 
either  degree  of  conformance  or  time  (e.g.,  are  we  on  schedule?).  When  these  occur,  operational 
assessors  will  troubleshoot  the  situation  to  locate  the  sources  of  the  shortfall  and  then  attempt  to 
diagnose  the  causes.  When  this  activity  has  been  successful  they  will  recommend  steps  that 
either  put  the  air  campaign  back  on  track  or  adapt  the  plan  to  better  fit  the  evolving  operational 
environment.  These  decisions  will  often  be  accompanied  by  allocation  recommendations.  These 
coordinated  decisions  on  the  part  of  operational  assessors  are  complex  and  interdependent.  It  is 
virtually  impossible  to  approach  any  degree  of  optimization  in  making  them.  Instead, 
operational  assessors  pursue  a  relaxed  criterion  throughout  this  process.  The  assessors  try  to 
make  decisions  that  are  “good  enough”  for  the  conditions  under  which  they  operate. 

4. 1 .5  Criteria  for  success 


Each  type  of  assessment,  therefore,  uses  different  criteria  for  success.  Tactical  assessment 
attempts  to  optimize  resources  and  maintain  strict  schedules  as  accurately  as  possible.  An 
important  success  measure  for  TA  is  the  number  of  targets  “serviced.”  This  includes 
maintaining  the  “service  levels”  of  these  targets  throughout  the  course  of  the  air  campaign. 
Tactical  assessors  also  strive  to  understand  local  effects.  The  primary  criterion  for  Operational 
Assessment  revolves  around  the  tradeoff  between  achieving  desired  effects  and  minimizing  the 
risks  of  that  achievement.  The  OAT  can  maintain  the  “right”  balance  in  this  tradeoff  by 
thoroughly  understanding  the  causal  structure  of  the  air  campaign  and  by  attaining  a  global 
comprehension  of  air  operations  and  their  relationships  to  the  overall  operation.  The  importance 
of  gaining  an  understanding  of  causal  structure  often  is  signaled  in  comments  from  operational 
assessors  like  “did  our  actions  cause  the  effect  I  am  seeing,”  “are  we  doing  the  right  things,”  and 
“are  we  doing  things  right.” 
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Behavioral  and  cognitive  differences  between  TA  and  OA  are  summarized  in  Table  1  below. 


Table  1 :  Summary  Comparison  of  TA  and  OA 


Tactical  Assessment 

Operational  Assessment 

Purpose 

•  Kinetic  &  non-kinetic 

•  Local 

•  L*  order 

•  Reconstitution 

•  Re-assignment 

•  Re-attack 

•  Kinetic  &  non-kinetic 

•  Global 

•  Nth  order 

•  Re-apportionment 

•  Re-allocation 

Objects  of 
Analysis 

•  Physical 

•  Single  systems 

•  Single  nodes  of  networks 

•  Physical  &  conceptual 

•  SoS  &  their  interactions 

•  Networks 

Workflow 

Differences 

•  Linear 

•  Strict  sequencing 

•  Synchronous 

•  Non-linear 

•  “Random  access” 

•  Asynchronous 

Types  of 
Decisions 

•  Sequencing/scheduling 

•  State  estimation 

•  Optimizing 

•  Assignment 

•  Plan  alterations 

•  Diagnosis/troubleshooting 

•  Satisficing 

•  Allocation 

Decision 

Processes 

•  Directed  search 

•  Confirmatory 

•  Constructive 

•  Opportunistic  search 

•  Diagnostic 

•  Integrative 

Criteria  for 
Success 

•  Resources  optimized 

•  Schedules  maintained 

•  Targets  neutralized 

•  Local  effects  understood 

•  Effects  produced 

•  Risk  minimized 

•  Causal  structure  understood 

•  Global  comprehension  of  ops 
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5.  Structure  of  the  OEAVT  Development  Team 


SAIC  assembled  a  multidisciplinary  team  to  carry  out  the  OEAVT  effort.  In  keeping  with  our 
general  development  methodology  for  joint  cognitive  systems,  disciplinary  participants  for  the 
team  were  defined  at  the  beginning  of  the  project  and  remained  throughout  the  project,  thereby 
ensuring  that  all  important  aspects  of  the  system  development  remained  in  the  forefront  from 
early  analysis  to  later  software  engineering.  The  SAIC  team  consisted  of  scientists  and  engineers 
in  cognitive  science,  systems  engineering,  software  engineering  and  assessment  operations. 
Additionally,  non-SAIC  team  members  included  experts  in  the  AOC  and  TBMCS,  cognitive 
system  engineers  involved  in  the  Phase  I  cognitive  task  analyses,  and  SMEs  possessing 
experience  in  operational  (and  tactical)  assessment.  In  keeping  with  the  nature  of  the  system 
under  development  and  the  work  being  supported,  the  project  was  led  by  a  cognitive  scientist. 

5.1  Subcontractors 


Each  of  the  subcontractors  on  the  OEAVT  project  provided  a  specific  contribution  to 
development.  Lockheed-Martin,  Inc.  brought  prior  AOC  and  specific  TBMCS  operational  and 
integration  experience  to  the  project.  The  ManTech  Cognitive  Systems  Engineering  Center 
(CSEC)  provided  continuity  from  the  “AOC  Strategy  Division  Decision  Analysis”  Phase  I  effort 
in  the  form  of  previous  Strategy  Division  cognitive  task  analyses.  They  also  produced 
visualization  concepts  for  indicator  management  that  were  incorporated  into  the  AECV  and 
module  of  OEAVT.  These  will  be  discussed  in  a  later  section  of  this  report.  Securboration, 
Incorporated  and  L3  Communications,  Incorporated  provided  Subject  Matter  Experts  throughout 
the  life  of  the  program.  Their  assistance  in  initial  analysis  of  operational  assessment,  formulation 
of  a  system  model  and  subsequent  requirements,  and  ongoing  evaluation  of  concepts  through 
several  developmental  stages  were  invaluable  in  keeping  the  development  team  on  the  right  track 
and  producing  the  right  product  for  the  assessment  task  and  the  warfighters  carrying  out 
assessment.  The  Securboration  SME  also  provided  access  to  the  Operational  Assessment 
community  and  served  as  liaison  to  the  Dynamic  Air  and  Space  Effects-based  Assessment 
(DASEA)  program. 
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6.  OEAVT  Program  Challenges 

There  were  three  areas  of  complexity  faced  by  the  OEAVT  program:  The  systems  of  record 
challenge,  integration  challenges  across  new  system  development  efforts,  and  the  general 
evolution  of  the  OA  concept. 

The  presence  of  TBMCS  and  IWPC  as  systems  of  record  in  the  AOC  led  to  challenges.  At  the 
outset  of  the  OEAVT  program,  it  was  assumed  that  the  visualization  capability  developed  by  the 
OEAVT  team  would  integrate  with,  or  otherwise  rely  on,  these  systems  of  record.  However, 
other  organizations  were  developing  potential  replacements  for  these  systems.  Thus,  there  was 
uncertainty  as  to  what  environment  OEAVT  would  have  to  interact  with  during  operations.  The 
IWPC  system  was  reaching  its  end  of  life  as  the  OEAVT  program  was  beginning.  However,  no 
replacement  had  been  selected  as  late  as  the  halfway  point  of  OEAVT.  This  left  the  visualization 
capability  of  OEAVT  in  need  of  a  computational  engine  to  carry  out  much  of  the  lower-level 
tasking  upon  which  assessment  depends;  including  search  functions,  some  data  fusion,  and 
tactical  level  computation  and  roll-up.  While  the  best  “vision”  for  success  would  have  been  to 
have  OEAVT  ride  “on  top  of’  such  a  computational  engine,  this  integration  was  not  achieved. 

Throughout  the  life  of  the  program  several  other  development  efforts  were  underway.  Each  of 
these  was  addressing  an  aspect  of  the  OA  problem,  sometimes  in  ways  complimentary  to 
OEAVT  and  sometimes  in  redundant  ways.  As  one  would  expect,  the  existence  of  these  separate 
efforts  posed  integration  challenges  for  a  complete,  seamless  solution  to  OA  support.  Of 
particular  significance  in  this  regard  was  the  development,  and  later  cancellation,  of  Theater 
Battle  Operations  Net-Centric  Environment  (TBONE).  TBONE  represented  the  future  integrated 
information  backbone  for  the  AOC.  With  the  cancellation  of  this  effort,  no  integrated 
information  capability  exists  within  the  Strategy  Division,  nor  is  one  expected  for  the  foreseeable 
future.  Since  TBONE  was  cancelled  in  the  middle  of  the  OEAVT  program,  a  shift  in  focus  was 
required,  due  to  the  loss  of  expected  information  services  from  TBMCS  1.1.3.  Instead,  a 
hodgepodge  of  disparate  systems  will  continue  to  be  a  significant  challenge  for  developers  and 
operators  alike.  As  a  result,  largely  manual  activities  supporting  operational  processes  will 
remain  the  norm.  The  fact  remains  that  realization  of  effective  and  efficient  operations 
absolutely  requires  automation  advances  far  beyond  what  we  are  seeing  today. 
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Finally,  the  evolution  of  the  OA  concept  has  continued  to  evolve  throughout  the  life  of  the 
OEAVT  program.  The  Strategy  &  Assessment  Requirements  Sub-Working  Group  was  created 
by  the  AFC2ISRC  prior  to  the  start  of  the  OEAVT  program.  It  has  served  as  the  main  interface 
between  Air  Combat  Command  (ACC)  and  the  OEAVT  program.  It  has  provided  a  forum  for 
discussing  assessment  requirements,  as  well  as  associated  issues  and  ideas.  In  addition,  it  has 
been  the  organizational  forum  for  bringing  together  warfighters  and  developers  on  a  regular 
basis.  A  challenge  that,  at  least  from  the  OEAVT  point  of  view,  was  not  addressed  during  the 
program  was  an  inability  to  fully  integrate  requirements  developed  by  different  groups  involved 
in  addressing  aspects  of  operational  assessment.  Requirements  for  OA  were  developed  from  at 
least  four  points  of  view,  across  as  many  organizations.  These  included  an  operational  point  of 
view,  a  systems  engineering  point  of  view,  a  software  point  of  view  and  a  cognitive  engineering 
point  of  view.  Unfortunately,  there  was  no  complete  integration  of  these  requirements. 
Subsequently,  system  developments  based  on  these  disparate  requirements  diverged  rather  than 
converging  toward  mutual  support. 
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7.  Description  of  Cognitive  Systems  Engineering  Methodology 
Underlying  OEAVT  Development 

The  SAIC  team  relied  on  an  adaptation  of  “conventional”  Cognitive  Systems  Engineering  (CSE) 
methods  and  processes  throughout  the  analysis,  design  and  development  phases  of  the  OEAVT 
program.  We  describe  the  methodology  in  this  section.  Our  application  of  the  methodology  to 
the  development  of  the  OEAVT  system  is  described  in  section  9. 

Our  methodology  is  based  on  the  firm  conviction  that  adequate  development  of  visualization  and 
decision  support  systems  must  be  guided  by  a  thorough  understanding  of  the  operational, 
behavioral  and  collaborative  factors  at  play  in  the  work  domain  being  addressed.  A  crucial 
concept  in  this  stance  is  that  the  to-be-developed  system  should  provide  adaptive  support  to  its 
users.  This  can  be  contrasted  with  more  traditional  approaches  that  strive  to  automate  critical 
work  processes  and/or  replace  humans  with  synthetic  “operators.”  The  OEAVT  team  adapted 
this  conventional  viewpoint  by  developing  a  methodology  that  integrated  cognitive  systems 
techniques  and  artifacts  with  rigorous  system  engineering  and  software  development  processes. 

7.1  Domain  analysis  and  derivation  of  perceptual/cognitive  requirements 


The  application  of  this  methodology  to  the  support  of  visual  thinking  begins  with  a  CSE  analysis. 
Our  analysis  typically  focuses  on  five  areas.  First,  a  work  domain  analysis  is  conducted  to  gather 
information  about  goals  and  requirements  present  in  the  domain  of  the  design,  work  constraints 
and  dependencies,  opportunities  or  affordances  utilized  by  workers,  and  the  functions  and 
physical  forms  of  current  technologies.  Second,  control  task  analysis  identifies  required 
activities,  tasks  and  workflow  relationships  inputs,  outputs,  and  knowledge  states  required  to 
support  successful  work  in  the  domain  of  interest.  Third,  we  identify  strategies  used  to  execute 
activities  and  tasks.  Our  focus  here  is  on  critical  cues  and  other  triggering  conditions  for  each  of 
the  tasks,  critical  decisions,  common  errors,  communication  patterns,  and  tools  used  to  carry  out 
the  work,  and  the  data  products  used  during  task  accomplishment.  Fourth,  we  identify  the  socio- 
technical  factors  present  in  the  domain.  Relevant  information  includes  organizational  structures, 
conditions  that  trigger  changes  to  these  structures  (e.g.,  conditions  creating  ad  hoc  teams),  and 
coordination/collaboration  dynamics.  Fifth,  we  identify  perceptual  and  cognitive  competencies 
required  for  success  in  the  domain.  We  characterize  these  as  perceptual  and  cognitive  work 
elements  used  to  manage  work,  achieve  associated  goals,  and  collaborate  with  teammates.  We 
collect  this  information  into  a  set  of  structured  concept  maps  containing  the  information 
enumerated  above. 
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7.2  Map  CSE  results  to  system  engineering  requirements  anaiysis  activities  and 
products 

A  thorough  CSE  analysis  is  a  necessary,  but  not  sufficient,  part  of  successful  system  design. 
Unless  the  information  gathered  during  CSE  analysis  is  meaningfully  integrated  with  system 
engineering  requirements  analysis,  then  it  remains  a  disconnected  data  set,  too  often  ignored  by 
system  developers.  We  avoid  this  problem  by  explicitly  mapping  CSE  analysis  results  to  system 
engineering  requirements  analysis  and  products.  We  define  1 1  system  engineering  requirements 
analysis  activities,  as  shown  in  the  center  section  of  Figure  3. 

Most  of  these  activities  constitute  functional  analysis.  However,  two  of  the  remaining  categories 
are  crucial  to  success  because  they  explicitly  define  both  the  goals  of  the  system  under  design 
and  constraints  that  will  come  into  play  during  the  design.  Cognitive  and  traditional  system 
engineers  participate  in  this  mapping  exercise,  explicitly  identifying  where  each  result  of  the 
CSE  analysis  will  be  placed  within  the  system  engineering  functional  analysis.  This  often 
requires  some  re-casting  of  terminology  and  level  of  specification  contained  in  the  CSE  analysis. 
This  is  desirable,  as  it  helps  to  translate  the  CSE  analysis  into  a  form  appropriate  for  system 
requirements  analysis  and  facilitates  a  close  working  partnership  between  the  CSE  analyst  and 
system  engineer.  Most  importantly,  performing  this  mapping  ensures  that  information  gathered 
during  the  CSE  analysis  is  included  in  the  system  engineering  requirements  analysis  products 
that  will  be  used  to  guide  subsequent  system  development. 
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Figure  3:  CSE  to  System  Engineering  Requirements  Mapping 
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7.3  Develop  a  system  model 


All  the  information  from  the  previous  analyses  then  is  used  to  develop  a  set  of  system 
engineering  models  in  CORE,  a  commercially  available  case  tool  that  allows  management  of  the 
entire  development  project  from  this  point  onward.  Working  with  SMEs,  we  develop  a  set  of 
hierarchically  organized  diagrams  of  the  work  domain  that  includes  information  about  system 
functionality,  inputs  and  outputs,  sequence,  constraints  and  dependencies,  triggering  conditions 
and  end-states  and  so  on.  Because  we  perform  the  CSE  -  system  engineering  analysis  mapping 
discussed  above,  we  can  be  confident  that  the  information  gathered  during  the  CSE  analysis  will 
be  included  in  the  system  engineering  model  developed  at  this  stage  of  the  design. 


7.4  Derive  system  requirements 

The  completed  CORE  diagrams  form  the  basis  of  requirements  identification.  System 
requirements  can  be  generated  automatically  by  the  CORE  system.  These  system  requirements 
then  are  augmented  with  requirements  identified  by  mapping  the  CORE-generated  set  against  the 
concept  maps  and  other  artifacts  of  the  CSE  analysis.  Each  requirement  is  labeled  with  a 
reference  to  its  source  and  color-coded  to  indicate  whether  it  is  a  system-only  requirement  or  a 
requirement  involving  user  participation  with  the  system  under  design. 


7.5  Map  requirements  to  perceptual/cognitive  work  elements 

We  can  be  confident,  at  this  point,  that  the  information  gathered  during  CSE  analysis  is 
represented  in  the  requirements  that  will  guide  system  design.  What  remains  is  the  crucial  step 
of  explicitly  relating  the  requirements  to  visualization  design.  We  accomplish  this  in  two  steps: 
first,  by  mapping  each  requirement  to  the  perceptual  and  cognitive  work  elements  that  will 
satisfy  it  and  second,  by  creating  an  executable  model  of  each  requirement.  We  discuss  the  first 
step  here.  The  second  step  will  be  discussed  in  the  next  section. 

Imagine  the  following  requirement  for  a  visualization  system  designed  to  support  Effects-Based 
Assessment  (EBA): 

The  system  shall  provide  a  way  to  derive  intended  and  unintended 
effects  from  tactical  assessment  results. 
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Clearly,  certain  competencies  must  be  used  to  satisfy  this  requirement.  We  identify  these  by 
referring  to  the  concept  maps  developed  during  CSE  analysis.  In  this  example,  the  requirement 
above  can  be  satisfied  with  a  combination  of  the  following  work  elements:  abduction  (inference 
to  the  best  explanation),  acquisition,  classification,  detection,  evaluation,  interpretation  and 
recognition.  Referring  to  the  concept  maps  we  enter  applicable  work  elements  into  a  spreadsheet 
opposite  a  list  of  the  system  requirements,  one  set  of  work  elements  for  each  requirement.  If 
there  are  elements  that  do  not  apply  to  any  requirement,  then  either  the  element  is  used 
unnecessarily  in  the  concept  maps  or  a  requirement  is  missing  from  the  system  engineering 
analysis.  The  concept  maps  thus  provide  the  database  containing  work  elements  required  to 
satisfy  requirements.  These  elements  will  usually  comprise  a  relatively  small  set  but  will,  in  the 
right  combinations,  successfully  address  a  large  collection  of  requirements.  Once  each 
requirement  has  been  assigned  one  or  more  perceptual  and  cognitive  elements,  we  are  ready  to 
begin  the  creation  of  executable  models. 


7.6  Visualization  Design 

Each  cognitive  and  perceptual  element  has  basic  visualization  requirements  defined  for  it  that  we 
assume  are  consistent  across  contexts.  For  example,  the  cognitive  element  compare  involves 
examining  two  or  more  objects  in  terms  of  their  similarities  and  differences.  The  visualization 
requirements  for  this  element  include  displaying  the  attributes  and  values  of  objects  being 
compared  to  users.  Further,  these  “objects”  should  be  displayed  in  a  way  that  facilitates  the 
comparison  being  carried  out,  that  is,  by  highlighting  the  similarities  and  differences  through 
some  method  of  ranking,  coding  or  some  other  means.  As  this  example  shows,  there  will  be  a 
basic  structure  associated  with  each  element  as  well  as  performance  parameters  for  the  elements. 
Parameters  for  compare  might  include  the  number  of  to-be-compared  elements  that  can  be  held 
in  short-term  memory  and  sensitivity  limitations  on  attribute  similarity  used  in  comparisons. 

The  requirements  provide  the  context,  constraints  and  boundaries  of  visualization  design  for  each 
primitive  element.  Thus,  while  the  basic  requirements  will  not  change  across  elements  the 
values  of  parameters  associated  with  modeling  of  elements  will  change  according  to  the  context 
of  each  requirement.  Consider,  for  example,  a  requirement  to  compare  an  air  attack  result 
against  a  target,  located  close  to  a  mosque,  with  the  intended  point  of  attack  to  assess  progress 
toward  an  effect.  In  this  case  the  sensitivity  parameter  for  the  comparison  would  be  set  to  a  high 
value,  since  collateral  damage  to  the  mosque  would  lower  the  assessment  of  success  toward 
effect.  The  comparison  of  planned  to  actual  result  would  indicate  success  only  if  the  attack  were 
extremely  precise,  that  is,  resulted  in  no  damage  to  the  mosque. 

By  this  method  we  develop  visualization  concepts  for  each  primitive  in  context.  Common 
requirement/primitive  combinations  are  collected  together  into  common  visualization  concepts. 
The  individual  concepts  then  are  aggregated  into  higher-level  collections  to  form  visualizations 
at  the  screen  level.  This  process  is  iterated  against  the  CORE  functional  model,  thereby  allowing 
validation  of  visualizations  by  ensuring  that  the  system  follows  the  processes  outlined  in  that 
model. 
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8.  OEAVT  OA  Cognitive  and  Work  Analysis 


The  process  used  in  developing  the  OEAVT  system  was  based  on  a  CSE  methodology  created 
from  the  system  engineering  and  integration  point  of  view  (as  outlined  above).  That  is,  CSE  was 
defined  to  be  an  integrated  member  in  an  overall  system  engineering  and  development  process 
leading  to  working  operational  systems.  This  is  in  contrast  to  a  philosophy  that  defines  CSE  as 
the  lead  discipline  in  system  development  efforts.  In  the  disciplinary  partnership  that  results 
from  the  former  philosophical  model,  CSE  analysis  strives  to  satisfy  two  objectives 
simultaneously.  This  first  goal  is  to  carry  out  the  analyses  needed  to  clearly  understand  the 
nature  of  the  work  involved  in  operational  assessment.  If  successful,  this  understanding  will 
include  organizational,  socio-technical  and  collaborative  processes  in  addition  to  the  more 
traditional  processes  involved  in  the  work  of  individual  operators.  The  second  goal  of  this  type 
of  CSE  analysis  is  to  gather  the  content,  and  communicate  that  content,  in  a  manner  that  can  be 
best  used  in  identifying  system  requirements  and  defining  a  system  model  that  includes  the 
important  CSE  considerations  uncovered  in  the  originating  analysis.  The  methodology  described 
below  satisfies  these  two  objectives. 

We  began  the  OEAVT  project  with  a  CSE  analysis  of  operational  assessment.  The  analysis  was 
based  on  four  sources  of  information.  Previous  analyses  had  been  conducted  in  an  earlier  phase 
of  the  program.  Although  these  analyses  focused  on  the  overall  processes  used  across  the 
Strategy  Division  rather  than  specifically  on  detailed  assessment  processes,  they  were  useful  as 
baseline  material  to  inform  our  analyses.  Operational  documents  formed  another  important 
source  of  information  for  our  analyses.  These  included  the  AFOTTP  that  was  currently  in  effect 
at  the  time  of  the  analyses,  briefings  and  instructional  materials  focused  on  operational 
assessment,  results  of  the  Air  Force  Assessment  Task  Force  (ATF)  meetings  on  definitions, 
processes,  lexicon  and  nomenclature  regarding  operational  assessment.  Our  third  information 
source  was  that  of  observations  and  interviews  conducted  at  operating  locations,  exercises  and 
working  meetings.  These  included  International  North  Atlantic  Treaty  Organization  (NATO) 
Exercises,  Joint  Expeditionary  Forces  Experiment  (JEFX)  and  the  April  2006  Warfighter 
Analysis  Workshop  (WAW).  The  final,  and  most  important,  source  of  information  was  that 
provided  by  the  SMEs  on  the  SAIC  development  team.  We  relied  on  two  SMEs  throughout  the 
life  of  the  project.  These  two  experts  were  uniquely  positioned  to  inform  our  analyses  of 
assessment  and  to  provide  ongoing  design  assistance  and  evaluation  throughout  development. 
One  of  these  SMEs  developed  and  taught  Operational  Assessment  at  the  705*  Command  and 
Control  Unit.  The  other  SME  was  the  Assessment  Team  Chief  at  the  8*  Air  Force  when  the 
OEAVT  project  began.  After  leaving  the  Air  Force  he  continued  to  play  a  crucial  role  in  the 
development  of  Operational  Assessment  concepts  on  several  programs  in  addition  to  OEAVT. 

Our  analysis  was  organized  according  to  the  five  areas  outlined  by  Vicente  (1999).  These 
included  a  work  domain  analysis,  control  task  analysis,  strategies  analysis,  socio-technical 
analysis  and  competencies/limitations  analysis.  We  documented  these  analyses  in  a  set  of 
concept  maps  that  were  organized  according  to  an  ontology  containing  information  in  each  of  the 
five  analytical  categories  enumerated  above.  The  concept  map  ontology  is  shown  in  Figure  4. 
Our  focus  in  the  work  domain  analysis  was  on  the  structure  and  logical  relationships  present  in 
the  operations  of  the  Strategy  Division,  with  emphasis  on  OAT  processes.  Our  work  domain 
analysis  focused  on  a  number  of  categories  important  to  the  accomplishment  of  assessment, 
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including  operational  requirements,  processes,  and  activities;  data  and  data  sources;  triggers,  pre¬ 
conditions  and  constraints;  products  and  artifacts. 


Figure  4:  Ontological  Categories  Used  in  the  OEAVT  CSE  Analysis 
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Control  tasks  were  the  focus  of  our  second  area  of  analysis.  Vicente  (1999)  defines  control  task 
analysis  as  the  identification  of  task  requirements  associated  with  known,  recurring  situations. 
Accordingly,  our  ontology  included  tasks  and  sub-tasks  as  elements  of  the  control  task  analysis. 
Another  crucial  element  of  the  control  task  analysis  was  to  identify  and  characterize  the  critical 
decisions  that  operators  made  during  the  unfolding  of  assessment  episodes.  We  used  the  critical 
decision  method  (Klein,  Calderwood  and  McGregor,  1989)  to  supplement  our  control  task 
analysis  with  these  characterizations.  Control  tasks  are  shown  as  salmon-colored  concepts  in 
Figure  4. 

The  control  task  analysis  was  followed  by  a  strategies  analysis.  Whereas  control  task  analysis 
addresses  the  question  of  what  needs  to  be  done,  strategies  analysis  asks  how  things  are  done. 

Our  main  emphasis  in  the  strategy  analysis  for  OEAVT  was  to  focus  on  behavioral  processes  and 
cognitive  work  elements  used  by  assessors  to  accomplish  the  control  tasks  required  for 
operational  assessment.  Behavioral  processes  exist  at  the  macro  level,  while  cognitive  work 
elements  are  micro-level  processes.  We  also  identified  the  information  and  information  sources 
used  to  carry  out  these  strategies;  the  goals,  constraints,  pre-conditions  and  contingencies  that 
affected  processes  and  work  elements;  and  the  errors  that  were  common  to  strategy  use.  The 
investigation  of  strategies  is,  arguably,  the  most  detailed  and  time-consuming  analysis  carried  out 
in  the  overall  CSE  investigation.  Program  constraints  often  will  limit  the  extent  of  this  level  of 
analysis,  particularly  at  the  Cognitive  Workflow  Elements  (CWEs)  level.  That  was  the  case  for 
the  OEAVT  program.  Thus,  our  cognitive  workflow  elements  analysis  was  largely  inferential  in 
nature.  That  is,  we  inferred  CWEs  from  the  behavioral  processes  identified  in  the  overall 
analyses  of  the  work  assessment  environment.  Our  CWE  analysis  was  then  halted  at  the  point  of 
identification,  and  we  moved  on  to  system  design  activities. 

Our  socio-technical  analysis  concentrated  on  the  systems  and  tools  with  which  assessors  must 
interact  and  on  the  organizations  and  teams  involved  in  assessment.  One  factor  that  is  not  often 
appreciated  about  the  AOC  environment  is  the  number  and  diversity  of  systems  that  play  a  role 
(both  positive  and  negative)  in  the  development  of  new  technology  like  OEAVT.  Legacy 
systems  and  evolving  technology  had  to  be  taken  into  consideration  during  the  design  of 
OEAVT.  Some  of  these  systems  represented  “services”  that  OEAVT  either  would  have  to  use  or 
might  benefit  from.  Others  represented  candidate  “hosts”  within  which  OEAVT  might  reside. 
Similarly,  tools  provided  both  beneficial  and  non-beneficial  influences.  Chief  among  these  was 
Microsoft  Excel,  used  by  many  of  the  assessment  teams  for  a  variety  of  tasks.  The  challenge  of 
this  tool  was  that  each  team  had  adapted  it  to  their  unique  needs  and  methodologies,  thereby 
posing  a  challenge  for  the  evolution  of  OEAVT  as  a  single  tool.  The  organizational  leg  of  our 
socio-technical  analysis  was  concentrated  on  the  formal  organizations  and  teams  involved  in  (we 
used  this  term  broadly)  assessment,  as  well  as  on  the  informal,  ad-hoc  teams  that  formed  and 
“de-formed”  as  assessment  progressed  over  time.  Organizational  information  that  we  identified 
included  formal  organizational  identity,  defined  and  ad-hoc  teams,  and  roles  played  by  these 
entities. 
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The  fifth  aspect  that  we  considered  was  that  of  capabilities  and  limitations  associated  with  the 
tasks  and  processes  of  assessment.  Normally,  such  an  analysis  would  consider  many  of  the 
cognitive  and  perceptual  limitations  that  are  common  in  the  human  factors  engineering  analysis 
of  performance  in  time-critical,  highly  risky  operations.  However,  time  and  resource  limitations 
on  the  OEAVT  design  team,  along  with  the  fact  that  assessment  is  not  as  sensitive  to  these 
performance  concerns,  led  us  to  relax  this  analysis.  Accordingly,  we  concentrated  only  on  time, 
information  requirements  and  complexity  in  our  capabilities  and  limitations  analysis. 

The  concept  maps  developed  in  the  early  stages  of  the  OEAVT  program  are  shown  in  Appendix 
A.  Fourteen  concept  maps  were  developed,  covering  pre-execution  and  execution  phases  of  the 
air  campaign.  As  can  be  seen,  several  of  the  concept  maps  apply  to  operational  processes 
normally  associated  with  tactical  assessment.  These  include  BDA,  functional  damage 
assessment,  physical  damage  assessment  and  mission  effects  assessment.  Their  presence  in  our 
analysis  indicates  that  the  separation  between  tactical  and  Operational  Assessment  was  not 
straightforward  at  the  time  the  concept  maps  were  developed.  In  fact,  both  of  the  SMEs 
participating  in  our  analysis  recommended  that  the  OEAVT  team  include  these  concept  maps  in 
our  overall  analysis.  That  they  are  not  developed  to  a  great  level  of  detail  indicates  (1)  the 
uncertainty  on  the  part  of  the  SMEs  about  where  the  line  between  tactical  and  Operational 
Assessment  should  be  drawn  and  (2)  the  feeling  on  the  part  of  the  design  team,  after  further 
document  review,  discussion  with  additional  operational  personnel  and  observation  of 
operational  exercises;  that  these  processes  did,  in  fact,  fall  more  on  the  tactical  side  of  the  line. 
The  remaining  concept  maps  in  Appendix  A  captured  our  analysis  of  Operational  Assessment  at 
varying  levels  of  detail  befitting  the  time  available  for  this  phase  of  analysis  and  the  perceived 
centrality  of  the  process  to  assessment  overall.  The  concept  maps  were  supplemented  with 
information  from  the  critical  decision  analysis.  This  information  is  presented  in  the  spreadsheets 
contained  in  Appendix  A.  In  these  spreadsheets,  the  critical  decisions  arising  from  the  Cognitive 
Work  Requirements  (CWR)  identified  in  Phase  I  were  enumerated  along  with  information 
requirements,  common  errors  and  other  information  needed  to  design  adequate  assessment 
support.  The  information  from  the  concept  maps  and  that  from  the  critical  decision  analyses 
were  combined  together  and;  along  with  documentation  from  operational  sources,  informal 
contacts  with  operational  personnel  and  SMEs  outside  the  project,  attendance  at  exercises  and 
other  operational  events,  and  participation  in  assessment  working  groups;  formed  the  basis  for 
system  engineering  analyses  and  requirements  development. 

Prior  to  the  development  of  system  requirements  the  design  team  identified  cognitive  workflow 
elements  (CWE)  needed  to  successfully  execute  assessment  strategies.  The  CWE  were 
identified  by  analyzing  the  control  task  and  strategy  information  in  each  of  the  concept  maps  and 
critical  decision  matrices.  Table  2  contains  the  CWE  that  were  identified  in  this  manner.  The 
purpose  of  this  analysis  was  to  identify  a  set  of  elements,  exhaustive  in  their  coverage  of 
assessment  that  could  be  used  to  create  a  traceability  matrix  relating  specific  design 
commitments  in  the  OEAVT  interfaces  and  operating  logic  to  the  system  requirements. 


24 


Table  2:  Cognitive  Workflow  Elements  (CWE)  Identified  from  the  CSE  Artifacts 


Acquire 

Communicate 

Discriminate 

Infer 

Monitor 

Aggregate 

Compare 

Estimate 

Integrate 

Plan 

Assign 

Decide 

Evaluate 

Identify 

Prioritize 

Choose 

Describe 

Generate 

Interpret 

Recognize 

Classify 

Detect 

Match 

Verify 
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9.  OEAVT  System  Engineering 


The  development  of  the  engineering  model  and  system  requirements  for  OEAVT  followed  a 
structured  process  based  in  traditional  systems  analysis  extended  to  include  the  products  of 
cognitive  analysis.  An  overview  of  the  process  is  illustrated  in  Figure  5.  Products  from  the 
systems  engineering  processes  (analysis,  interviews,  models,  and  requirements)  were  captured  in 
the  Systems  Engineering  tool  CORE. 


Functional  Representation 
(CORE) 


Figure  5:  System  Analysis  Process 
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9.1  Step  1 :  Perform  OEAVT  Concept  Map  Analysis 


The  first  step  in  the  proeess  was  to  perform  coneept  map  analysis.  Seetion  9  deseribes  the 
proeess  that  the  OEAVT  team  used  to  develop  the  concept  maps  for  the  Operational  Assessment 
activities  in  the  AOC  and  provides  a  detailed  description  of  the  development  and  analysis 
process  and  activities.  The  output  of  the  concept  map  analysis  included  both  the  ManTech  Spiral 
1  and  2  cognitive  engineering  analysis  products  (such  as  Goal  Process  Nodes  (GPN),  Process 
Steps  (PS),  Cognitive  Work  Requirements  (CWR)  and  the  Information  Relationship 
Requirements  (IRR))  and  a  set  of  spreadsheets  that  contained  engineering  requirements 
developed  using  the  concept  map  analysis.  The  CORE  database  schema  had  to  be  extended  to  so 
that  the  GPN’s,  PS’s,  CWR’s,  IRR’s,  and  workflow  information  could  be  stored  and  the 
relationship  between  these  specific  attributes  could  be  represented  in  CORE.  The  document  or 
analysis  source(s)  of  each  CORE  entry  were  logged  permitting  attribution  for  each  cognitive 
analysis  product.  Figure  6  (below)  illustrates  a  sample  CORE  page  showing  a  GPN,  the  source 
of  the  data,  and  supporting  relationships. 


3  CORE  Database  Homepage  -  Microsoft  internet  Explore 


I  File  Edit  View  Favorites  Tools  Help 


B0B 


CORE  Database  for  Project:  OEAVT 


Select  one  of  the  following  Class  or  Folder 
links  to  get  a  list  of  local  elements 


Activities 


Cognitivie  Work  Requirement 


CompletionCriterion 


Constaints  S:  Dependencies 


CWR 

Data 

Data  Sources 

Document 

Engineer 

ExternalGraphic 


Function 

■Goal  Process  Node 

jimplication 

! Information  Relationship 

j  Requirements 

iiRR 


'Issue 

|ltem 


jParticipants 
jprocess  Step 

Products 

jWorkflow 


Goal  Process  Node:  3.2  Successfully  Infer  Adversary  Goals 


Abbreviation 

Created 

Creator 


Description 


Last  Modified 


Monday;  December  13;  2004  at  02:09:58  PM 
Overdorf 

The  purpose  of  GPN  3.2  (Infer  Adversary  Goals)  is  to  develop 
and  maintain  an  understanding  of  adversary  end-states  that 
may  impact  Aerospace  Objectives  (GPN  1.1).  GPN  3.2  (Infer 
Adversary  Goals),  thus,  supports  the  selection  of  Aerospace 
Objectives  (GPN  1.1),  by  providing  information  about  end- 
states  the  coalition  believes  the  adversary  desires  (or  should 
desire).  The  identification  of  Adversary  Goals  is  supported  by 
Posited  Adversary  Capabilities  (GPN  3.4)  and  inferences 
about  Adversary  Will  (GPN  3.6).  Propositions  about 
Adversary  Capabilities  foster  identification  of  the  adversary's 
dangerous  and  likely  goals.  Inferences  about  Adversary  Will 
help  to  ensure  that  Inferred  Adversary  Goals  are  not 
constrained  by  assumptions  of  a  rational  adversary.  Because 
GPN  3.2  produces  inferences  about  Adversary  Goals  (only  the 
adversary  knows  their  goals  with  certainty),  inference 
validity  is  a  constant  concern.  As  a  result,  the  inputs  that 
support  inferences  about  Adversary  Goals  must  be  monitored 
continuously.  Indicators  (GPN  4.4)  support  the  continuous 
assessment  of  Inferred  Adversary  Goals. 

Tuesday;  December  14;  2004  at  03:08:36  PM 


Number 

3.2 

Relationships  1 

Achieved  By 

Process  Step  3. 2. a  Potential  Adversary  Goals 

Process  Steo  3.2.B  Identify  Relevant  Adversary  Goals 

Process  Step  3.2.c  Inferred  Adversary  Goals 

Process  Steo  3.2.d  Assess  Adversary  Goals 

Process  Step  3.2.e  Assessed  Adversary  Goals 

documented  by 

Document  2  Spiral  2  -Aerospace  Command&Control  Cognitive  Analysis 

Monitored  By 


owned  by 
reveals 


Cognitivie  Work  Requirement  3. 2.x. 2  Evaluate  quality  of  inferred  adversary 

goal  set 

Cognitivie  Work  Requirement  3. 2.x. 3  Monitor  extent  to  which  inferred 

adversary  goal  set  provides  information  for  the  development  of  aerospace 

Cognitivie  Work  Requirement  3. 2.x. 4  Assess  potential  for  inferred  adversary 

goals  to  impact  aerospace  objectives 

Engineer  Overdorf 
Implication  GPN  3.2  Implications 


Figure  6:  CORE  Database 
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9.2  Step  2:  Develop  Functional  System  Representation  and  Analysis 


The  next  step  in  the  process  was  to  develop  a  model  of  the  Operational  Assessment  process 
within  the  AOC.  Air  Force  documentation  was  examined  to  discover  the  documented  (or 
expected  way)  that  Operational  Assessment  was  to  be  conducted.  AFTTP  2-3.2  was  used 
extensively  to  identify  tasks/activities  that  were  required  in  the  Operational  Assessment  process, 
the  sequencing  of  these  tasks,  inputs  to  the  tasks  and  outputs  from  the  tasks.  Activity  networks 
were  developed  in  a  “top  down”  manner  starting  with  high-level  tasks  or  activities.  Figure  7 
(below)  illustrates  the  top  level  AOC  functional  description  of  operational  assessment.  Each  of 
the  top-level  tasks  was  then  further  decomposed  to  reveal  progressively  lower  level  assessment 
operations.  Figure  8  reveals  a  decomposition  of  Operational  Assessment  during  the  execution 
phase  of  the  conflict.  At  each  level  of  decomposition,  information/products  required  for  the  task, 
and  information/  products  produced  by  a  task  were  documented.  Figure  9  shows  a 
decomposition  of  the  “Analyze  Operational  Results”  task  and  the  associated  task  inputs,  outputs 
and  task  triggers.  CORE  supports  the  network  representation  (and  output)  in  a  variety  of 
notations  (FFDB,  enhanced  FFDB,  IDEFO,  and  N2).  Different  views  were  used  depending  on  if 
they  enlightened  activity  relationships  or  better-revealed  sequencing.  The  source  of  the 
information  for  the  identified  tasks  was  traced  to  the  information  source  in  the  “documented  by” 
field. 


Figure  7:  Top  Level  Operational  Assessment  Task  Diagram 
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Figure  8:  Decomposition  of  the  “Execution”  Operational  Task 


3  CORE  Database  Homepage  -  Microsoft  Internet  Explorer 


I ;  File  Edit  Vtew  Favorites  ^ois 


CORE  Database  for  Project:  OEAVT 


Analyze  Operational  Results  Enhanced  FFBD 

Element  View:  |  Enhanced  FFBD 


Figure  9:  “Analyze  Operational  Results”  Decomposition 
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The  OEAVT  model  is  documented  as  a  hardcopy  set  of  Enhanced  Functional  Flow  Block 
Diagrams  generated  by  CORE  and  a  set  of  interactive  “html”  browser  forms.  Appendix  B 
contains  the  hardcopy  set  of  model  diagrams. 

9.3  Step  3:  SME  Review  of  Functional  Representation 


The  next  step  in  the  process  was  to  review  the  “doctrinally  correct”  Operational  Assessment 
process  network  with  SMEs  from  the  assessment  community.  Over  a  series  of  sessions,  the 
SMEs  reviewed  each  task,  the  inputs/outputs,  and  the  sequencing  from  the  drawing.  Often,  they 
provided  additional  details  that  permitted  the  network  to  be  revised  and  corrected  and  the  process 
better  understood.  Occasionally,  they  provided  insight  into  differences  between  Operational 
Assessment  as  a  concept  and  Operational  Assessment  as  practiced  in  the  AOC.  Each  change  by 
an  SME  was  noted  in  CORE  and  attributed  to  the  particular  SME.  At  the  completion  of  this  step, 
a  baseline  model  of  Operational  Assessment  was  in-place. 

9.4  Step  4:  Map  Functional  and  Cognitive  Requirements 


The  baseline  model  of  Operational  Assessment  highlighted  what  must  be  done  to  perform 
Operational  Assessment  in  the  AOC.  The  next  step  in  the  process  involved  the  creation  of  a  set 
of  system  requirements  for  OEAVT  that  would  permit  it  to  support  all  of  the  OA  tasks  on  at  least 
some  level.  Each  function  in  the  network  was  examined  to  determine  if,  and  how,  OEAVT 
might  support  the  function.  The  OEAVT  tool  was  assigned  a  set  of  system  requirements  for 
each  function  that  provided  the  specified  level  of  support.  These  requirement  assignments  were 
broadly  scoped  and  targeted  for  a  “large  scale”  OEAVT  implementation  encompassing  almost  all 
assessment  tasks.  Layered  on  these  requirements,  were  the  cognitive  requirements  developed  in 
the  concept  map  analysis  associated  with  cognitive  work,  information,  and  workflow.  Each 
requirement  was  entered  as  an  “Originating  Requiremenf  ’  (CORE  name)  and  linked  to  the 
function  and  related  analysis.  Figure  10  illustrates  an  example  of  a  function  (“RFI  from  ISRD”) 
and  shows  the  functional  hierarchy,  supporting  documents,  outputs,  and  originating 
requirements. 
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Figure  10:  Core  Example  of  a  Function  and  the  Associated  Originating 

Requirements 


Figure  11  shows  two  of  the  originating  requirements  for  function  “RFI  from  ISRJD”.  Note  that 
each  originating  requirement  traces  to  multiple  functions  and  that  the  source  of  the  requirement 
is  documented. 
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Figure  11:  Example  of  Two  OEAVT  Requirements 


9.5  Step  5:  Scope  Requirements 


The  final  step  in  the  process  was  scoping  the  requirements.  The  model  created  by  the  process 
included  the  OA  tasks  for  all  phases  of  a  conflict.  It  was  obvious,  early  on,  that  it  would  not  be 
possible  to  support  all  of  Operational  Assessment  with  the  OEAVT  tool.  Rather  than  decide,  a- 
priori,  what  Operational  Assessment  tasks  OEAVT  would  perform,  it  was  decided  to  look  at  the 
tasks  that  OA  needed  to  perform  and  assign  the  most  relevant  ones  to  OEAVT.  The  scoping  task 
involved  an  examination  of  each  of  the  OA  functions  and  the  associated  originating 
requirements.  The  OEAVT  team  conducted  internal  reviews  of  the  desired  functionality, 
compared  the  desired  functionality  with  the  statement  of  work,  and  prioritized  choices  based  on 
discussions  with  the  OA  SMEs.  Candidate  lists  of  requirements  to  eliminate  were  created. 

These  lists  were  formally  reviewed  and  iterated  with  the  customer  until  a  final  list  of 
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requirements  was  developed.  The  final  list  of  requirements  is  documented  in  Appendix  C.  Note 
that  the  appendix  includes  all  of  the  334  original  requirements  and  that  a  number  of  these 
requirements  “generated”  issues  (CORE  column  name).  The  issues  in  the  “generates”  column 
document  the  disposition  of  the  original  requirement.  A  blank  in  the  “generates”  field,  indicates 
that  the  requirement  was  accepted.  Other  issues  include:  “out  of  scope”,  defective,  and 
“potential”  requirements.  Requirements  with  these  issues  were  ultimately  rejected.  Other 
“generates”  issues  resulted  in  changed,  but  ultimately  accepted,  requirements. 
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10.  OEAVT  SOFTWARE  ARCHITECTURE 


10.1  Software  Engineering  Overview 


The  OEAVT  Software  Engineering  effort  was  a  rigorous  process  conforming  to  the  OEAVT 
Project  Management  Plan  and  executed  in  compliance  with  CMMI^”^  Level  3  processes.  The 
OEAVT  Software  Implementation  is  to  be  used  in  Air  Operations  Center  (AOC),  targeting  the 
Operational  Assessment  Team  (OAT)  as  the  principle  user.  From  the  outset,  the  OEAVT 
visualizations  were  designed  as  separable/stand-alone  components  subscribing  to  a  common  data 
model  (using  a  publication/subscription  mechanism)  and  communicating  via  anonymous 
controller  events  utilizing  the  Model-View-Controller  (MVC)  design  pattern^  (with  particular 
emphasis  on  the  Java  implementation"^). 

The  final  instantiation  of  the  OEAVT  Software  Implementation  was  the  System  for  Cognitive 
Visualization  of  Operational  Assessment  (SCVOA).  The  SCVOA  was  a  stand-alone  application 
designed  to  demonstrate  the  OEAVT  visualizations.  The  original  goal  of  integration  with  AOC 
systems  of  record  was  not  realized,  primarily  because  of  program  direction  decisions  made 
outside  of  OEAVT  and  AFRL/RH.  These  included  cancellation  of  the  TBONE  program  and  the 
decision  to  “fast  track”  other  systems  for  fielding  in  the  AOC.  The  SCVOA  software  design, 
however,  has  been  optimized  for  interfacing  with  future  AOC  systems  of  record  once  they  have 
been  fielded. 

The  SCVOA  incorporated  most  of  the  major  assessment  concepts  envisaged  during  the 
investigatory  phase  of  the  project  including  elements  of  cognitive  workflow.  Central  to  the 
application  workflow  was  the  concept  of  perspectives.  The  notion  of  “perspectives^”  in  the 
generic  sense  is  not  new.  However,  the  SCVOA  was  the  first  application  we  know  of  that 
implemented  assessment  perspectives.  We  used  assessment  perspectives  to  help  manage 
cognitive  workflow  in  the  application. 

The  SCVOA  has  four  defined  perspectives:  Action-Effect  Contrast  Visualization  (AECV),  Geo¬ 
spatial  Visualization  (GEV),  Indicator  Analysis  Manager  (lAM),  and  the  Assessment  Task 
Manager  (ATM).  Of  the  four  perspectives,  only  the  AECV  perspective  was  fully  implemented 
due  to  time  and  budget  constraints.  The  GEV  module  was  removed  from  OEAVT  after  the 
planning  and  assessment  contractor  development  teams  determined  that  it  was  redundant  to  a 
similar  capability  contained  within  the  planning  system.  Again,  this  system  could  have  utility  in 
supporting  some  aspects  of  assessment,  if  the  operational  community  decided  that  geo-spatial 
information  was  a  desirable  feature  in  an  assessment  support  system.  The  other  two  perspectives 
were  populated  with  static  concepts. 


3  Applications  Programming  in  Smalltalk-80™:  How  to  use  Model- View-Controller  (MVC).  Steve  Burbeck, 
Ph.D,  1987,  1992.  Online,  University  of  Illinois  at  Urbana-Champaign.  (12  Aug,  2008).  http://st- 
www.cs.uiuc .  edu/users/smarch/st-docs/mvc  .html 

4  Java  Blueprints  -  J2EE  Patterns.  Sun  Microsystems,  Inc.  Online.  (12  Aug,  2008). 
http  ://j  ava.  sun.  com/blueprints/pattems/MV  C-detailed.html 

5  Using  Perspectives  in  the  Eclipse  UI.  Object  Technology  International,  Inc.,  2001.  Online,  Eclipse.org.  (13 
Aug,  2008).  http://www.eclipse.org/articles/using-perspectives/PerspectiveArticle.html 
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10.2  Software  Components 


The  SCVOA  is  a  pure  Java  implementation,  making  use  of  three  open-souree  libraries  (in  the 
form  of  Java  JAR  files)  in  addition  to  the  standard  libraries  available  in  the  Java  J2SE  1 .4.2 
environment.  The  three  open-souree  libraries  are: 

•  JFreeChart6  (jeommon.jar,  jfreechart.jar)  -  used  to  implement  all  bar  eharts  (including 
the  AECV  itself)  and  the  spider  web  plot. 

•  Apache  Batik?  (batik-awt-util.jar)  -  used  to  generate  the  gradient  in  the  AECV. 

•  H2  Databases  (h2.jar)  -  used  to  implement  persistent  data  storage  (used  by  the  Persistent 
Data  Store  [PDS]  package). 

The  SCVOA  itself  is  comprised  of  four  major  Java  packages  as  shown  in  Figure  12.  The  top- 
level  package  has  the  fully  qualified  name  of  “mil.af  afrl.hec.”  Directly  underneath  the  top-level 
package  are  the  four  primary  software  components: 

1 .  mil.af  afrl.hec. aecv  -  contains  all  of  the  AECV-specific  source  modules 

2.  mil.af  afrl.hec. awm  -  contains  all  of  the  AWM-specific  source  modules 

3.  mil.af  afrl.hec.gev  -  contains  all  of  the  GEV-specific  source  modules  (currently 
unimplemented) . 

4.  mil.af  afrl.hec. scvoa  -  contains  all  of  the  SCVOA-specific  source  modules,  including 
the  “common”  modules. 


6  JfreeChart.  Object  Refinery  Limited,  2005-2008.  Online.  (13  Aug,  2008).  http://www.jfree.org/jfreechart/ 

7  Batik  SVG  Toolkit.  The  Apache  Software  Foundation,  2000-2008.  Online.  (13  Aug  2008). 
http://xmlgraphics.apache.org/batik/ 

8  H2  Database  Engine.  The  H2  Group.  Online.  (13  Aug,  2008).  http://h2database.com/ 
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Figure  12:  SCVOA  Package  Overview 

The  packages  and  modules  in  mil.af.afrl.hec.scvoa.common  are  “common”  to  all  packages  in  the 
SCVOA  (intimated  by  the  dashed-lines  in  Figure  12).  The  SCVOA  is  explicitly  designed  to  be 
separable  at  the  package  level.  For  example,  the  aecv  package  has  no  dependencies  upon  (i.e. 
does  not  require  in  order  to  compile  or  be  instantiated)  the  awm  package  or  the  plan  hierarchy 
display  {phd)  package.  However,  the  aecv,  awm,  and phd  packages  depend  upon  the 
scvoa.  common  package. 

The  scvoa.common  package  contains  all  of  the  components  of  the  data  model.  The 
scvoa.common.ee  sub-package  contains  all  of  the  classes  and  interfaces  needed  to  support 
Controller  Events.  The  scvoa.common.pd  sub-package  contains  all  of  the  classes  and  interfaces 
needed  to  support  Persistent  Data  publication  and  subscription.  These  three  packages  are 
required  by  just  about  every  component  in  the  SCVOA. 


10.3  SCVOA  Application 


The  implementation  of  the  SCVOA  application  itself  can  be  found  in  the  scvoa. app  package. 
Due  to  its  demonstrative  nature,  this  package  has  dependencies  upon  all  of  the  other  packages. 
The  application  package  encapsulates  all  of  the  user  interface  logic  in  the  main  window, 
including  the  perspective  software  implementation  and  the  day  controller. 
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10.3.1  Demonstration  Mode 


One  of  the  unique  features  of  the  SC  VO  A  is  the  ability  to  restart  the  application  and  initialize  as 
if  it  were  at  a  different  point  in  time  than  “the  current  real  time.”  This  facilitates  demonstrations 
of  the  application  by  showing  the  assessment  at  one  point  in  the  conflict,  then  “fast-forwarding  in 
time”  to  see  how  the  assessment  has  changed  later  in  the  conflict. 

This  feature  was  implemented  by  engineering  the  application  so  that  each  scenario  in  the 
persistent  data  store  (PDS)  has  a  single  absolute  time  reference  for  the  start  of  the  conflict  (i.e., 
“D-0”).  All  other  time  references  in  the  PDS  are  relative  to  D-0.  By  pushing  D-0  back  in  time, 
the  assessor  effectively  moves  forward  in  simulated  time.  This  is  all  handled  quite  transparently 
by  the  SCVOA. 

10.3.2  Demonstration  Data 


As  the  data  model  solidified,  it  became  clear  that  the  SCVOA  would  require  a  significant  amount 
of  data  in  order  to  display  something  meaningful.  Clearly  this  kind  of  data  would  be  available  in 
a  real  military  conflict  such  as  one  the  tool  was  designed  for.  The  question  was,  “How  do  we 
demonstrate  and  test?” 

To  that  end,  we  created  an  unclassified  sample  scenario  with  sufficiently  detailed  data  to  show 
both  how  the  application  works  and  why  it  is  relevant.  As  a  happy  aside,  it  provided  most  of  the 
conditions  we  required  in  order  to  perform  acceptance  testing. 

Normally,  the  initial  data  for  the  system  would  be  acquired  via  machine-to-machine  interfaces 
from  AOC  systems  of  record.  We  were  constrained  by  the  need  to  have  an  unclassified  data  set. 
Rather  than  write  an  entire  user  interface  for  our  data  entry  that  would  be  superfluous  in  the  real 
AOC  environment,  we  chose  to  create  the  data  using  a  spreadsheet  and  import  the  data  as 
comma-separated  value  (CSV)  files. 

Attaining  data  consistency  and  coherence  with  a  spreadsheet  proved  to  be  challenging.  Initial 
inconsistencies  were  often  revealed  during  the  import  process  by  a  database  constraint  failure. 
Further  data  checks  were  added  to  the  data  import  mechanism  to  enhance  data  entry  validation. 
As  the  tool  matured,  the  visualizations  themselves  proved  to  be  of  significant  value  in  ferreting 
out  the  last  data  inconsistencies. 

10.3.3  Plan  Hierarchy  Visualization 


The  scvoa.phd  package  (hereafter  referred  to  as  the  “PHD”)  encapsulates  the  Plan  Hierarchy 
Visualization.  This  visualization  is  JTree  implementation  with  Controller  Event  generation  and 
Persistent  Data  Subscription  to  the  Campaign  Plan  and  Effects.  The  PHD  is  a  good  example  of 
the  minimal  implementation  needed  for  a  functional  SCVOA  component. 
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10.3.4  Assessment  Visualizations 


All  of  the  supporting  assessment  visualizations  (except  for  the  AECV  itself)  are  encapsulated  in 
the  mil.af.afrl.hec.awm  package.  This  includes: 

•  Effect  Attribute  Visualization 

•  Indicator  Attribute  Visualization 

•  Indicator  List  (including  the  IWPC  eCAT  Spider  Diagram) 

•  Threshold  Visualization 

•  Weight  Visualization 

•  Multi-Thumb  Color  Slider 

10.3.5  Value  Model 


The  Value  Model  implementation  can  also  be  found  in  the  mil.af.afrl.hec.awm  package.  The 
SCVOA  implements  the  Value  Model  as  its  own  thread,  running  on  demand  in  response  to  a 
significant  change  in  the  data  model.  During  first  invocation,  the  Value  Model  traverses  the 
entire  Effect  Hierarchy  once,  bringing  all  assessments  up-to-date  as  needed.  For  large  effect 
hierarchies,  this  initial  assessment  may  take  a  perceptible  amount  of  time. 

The  Value  Model  provides  the  assessor  with  a  quantitative  assessment  of  the  effects-based  plan. 
The  following  discussion  summarizes  the  concepts. 

Central  to  the  Value  Model  is  the  comparison  of  effort  expended  with  assessed  effectiveness  in 
the  form  of  a  truth  table  Figure  13. 
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Figure  13:  Greathouse  Truth  Table 
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As  a  "two-by-two"  table,  there  are  four  conditions:  two  that  are  normal  (i.e.  they  conform  to 
expectations)  and  two  that  signal  a  condition  requiring  investigation.  The  conditions  are 
expressed  in  the  form  of  effort  expended/effectiveness  assessed.  True  and  false  suggest  extreme 
limits  in  the  existence  of  an  effort  expended  and  an  effect  to  be  assessed. 

1.  False/False  -  no  effort  expended,  no  assessed  effect.  This  condition  is  to  be 
expected  and  does  not  particularly  merit  additional  investigation. 

2.  True/True  -  effort  expended  and  a  corresponding  level  of  effectiveness  is  noted. 
This  condition  is  also  what  the  assessor  expects  and  is  not  an  indication  of 
something  amiss. 

3.  True/False  -  effort  expended,  but  no  assessed  effectiveness.  This  condition 
requires  further  investigation  into  why  the  effect  is  not  being  achieved. 

4.  False/True  -  no  effort  expended,  but  a  perceived  effectiveness  has  been  assessed. 
This  also  bears  further  investigation  into  the  underlying  data  supporting  the 
assessment.  In  this  case,  the  apparent  effectiveness  is  suspect  until  an  explanation 
has  been  deduced. 

Conversion  of  these  two  input  conditions  from  boolean  values  to  rational  numbers  allows  an 
assessment  to  be  quantified  within  a  continuous  assessment  space. 

However,  a  scale  over  which  effort  and  effectiveness  can  be  evaluated  must  be  determined.  The 
OEAVT  team  chose  to  quantify  performance  in  terms  of  DMPI-Sortie-Equivalents  (DSEs).  {For 
an  explanation  of  DMPIs  and  DSEs  as  applied  to  kinetic  planning  and  assessment,  see  AFTTO 
3-3.AOC,  1  November  2007  FINAL,  section  3. 3. 3. 3).  A  DSE  table  was  built  with  assigned  DSEs 
for  specific  combinations  of  delivery  platform  and  ordnance.  This  document  postulates  ways  to 
describe  non-kinetic  effort  (e.g.  refueling  sorties,  PsyOps,  etc.)  in  terms  of  DSEs. 

Assessed  effectiveness  is,  in  the  end,  a  subjective  evaluation  of  an  effect  based  on  established 
assessment  criteria  (Indicators)  and  the  weight  assigned  to  those  criteria.  The  Value  Model 
allows  the  assessor  to  transform  that  evaluation  into  a  continuous  assessment  space  with 
objective  results  in  the  form  of  the  Value  Model  roll-up  (see  Figure  14  below). 
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Figure  14:  Value  Model  Roll-up 

The  primary  calculation  in  the  Value  Model  is  a  summation  of  products:  weight  times  Indicator 
assessed  effectiveness.  An  assumption  of  the  Value  Model  is  that  all  the  weights  in  the 
calculation  must  add  up  to  one  (1).  The  OEAVT  value  model  is  a  weighted  mean,  with  the 
requirement  that  all  of  the  weigh  coefficients  are  positive  and  sum  to  one.  (Note  that  this  special 
case  of  a  weighted  mean  with  positive  weight  coefficients  that  sum  to  one  is  also  known  as  a 
convex  combination). 

As  the  diagram  shows,  the  Value  Model  is  analogous  to  a  tree  in  the  sense  that  the  leaves  of  the 
tree  are  the  Indicators  associated  with  the  Tactical  Tasks  in  the  Campaign  Plan.  The  Value 
Model  calculation  for  Tactical  Task  assessed  effectiveness  is  a  simple  summation  of  products  for 
that  Tactical  Task's  Indicators. 

Continuing  the  tree  analogy,  the  twigs  in  the  tree  are  the  rolled-up  assessed  values  of  the  Tactical 
Tasks.  In  this  analogy,  the  Tactical  Objectives  would  be  the  small  branches  in  the  tree.  The  roll¬ 
up  for  the  Tactical  Objectives  is  complicated  by  the  fact  that  a  Tactical  Objective  may  (and 
probably  will)  have  associated  Indicators  in  addition  to  its  child  Tactical  Tasks.  The  Value 
Model  must  take  into  account  both  (Indicators  and  Tactical  Tasks)  in  the  roll-up  calculation  for 
the  Tactical  Objective. 
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The  weights  of  the  associated  Indicators  and  the  child  Tactical  Tasks  must  add  up  to  one  (1). 

The  Operational  Objectives  (the  major  branches  in  the  tree)  roll-up  in  the  same  manner  as  the 
Tactical  Objectives,  taking  into  account  both  their  own  Indicators  and  their  child  Tactical 
Objectives. 

The  assessor  is  responsible  for  updating  or  adding  new  Indicator  values  based  on  new 
tactical/operational  information  and  adjusting  the  weights  as  the  dynamics  of  the  conflict  change. 
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11.  OEAVT  System  Components 

Four  major  components  were  developed  for  the  OEAVT  system  during  the  life  of  the  program. 
Not  all  of  these  components  reside  in  the  eurrent  version  of  the  system.  One  challenge  that  the 
development  team  had  throughout  the  life  of  the  program  was  the  dynamic  nature  of  the 
operational  environment.  This  led  to  changing  requirements  and,  in  some  eases,  re-direetions 
based  on  program  deeisions  at  the  ACC  level. 

One  example  of  this  was  the  development,  and  subsequent  discarding,  of  the  (TI3). 

Development  was  begun  when  it  seemed  increasingly  clear  that  TBMCS  would  be  replaeed  with 
TBONE.  The  TI3  was  then  put  into  development  in  order  to  acquire  the  strike  results  from 
TBONE  that  would  be  needed  to  “feed”  the  AECV.  When  the  TBONE  effort  was  subsequently 
cancelled  the  TI3  eomponent  was  discarded,  although  this  module  eould  still  be  extremely 
valuable  in  supporting  the  OAT’s  need  to  aequire  strike  results  for  purposes  of  assessment. 

Another  development  was  the  assessment  visualizations  for  the  Global  Effeets  Matrix- 
Synehronization  (GEM-S).  This  effort  was  to  improve  initial  visualization  coneepts  developed 
by  personnel  of  the  Joint  Information  Operations  Warfare  Center  (JIOWC). 


11.1  Action-Effect  Contrast  Visualization 


11.1.1  Purpose 


The  AECV  system  is  the  eenterpieee  of  OEAVT.  It  supports  the  two  primary  funetions  of 
Operational  Assessment.  These  are  the  assessments  of  progress  toward  desired  effects;  and  the 
management  and  evaluation  of  indieators,  evidence  and  tactical  assessments  upon  which 
Operational  Assessments  are  based.  These  funetions  stood  at  the  top  of  an  operational 
requirements  hierarchy  that  emerged  during  the  early  portions  of  the  program  from  operational 
documents,  researeh  doeuments,  observations  of  field  operations  and  interviews  with  assessment 
SMEs.  Previous  analyses  of  the  assessment  environment  also  reinforeed  the  existence  of  these 
funetions  as  two  of  the  three  crucial  aspects  of  successful  assessment.  Based  on  the  operational, 
eognitive  and  system  analyses  earned  out  in  early  projeet  stages,  the  OEAVT  team  decomposed 
these  two  high-level  funetions  into  several  component  funetions.  This  decomposition  formed  the 
basis  for  the  development  of  the  system  requirements  upon  whieh  specific  capabilities  would  be 
developed.  Eaeh  is  briefly  introdueed  below  in  the  seetion  on  Current  Status. 
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11 .1 .2  Development  Process 


The  AECV  was  an  outgrowth  of  a  general  work  domain  and  cognitive  analysis  of  Operational 
Assessment  carried  out  in  the  initial  stages  of  the  OEAVT  program.  These  analyses  were 
themselves,  refinements  of  analyses  of  the  Strategy  Division  carried  out  in  Phase  I  of  the  RH  for 
AOC  program.  The  Phase  I  analyses  were  carried  out  by  ManTech  Corporation  and  focused  on 
activities  carried  out  across  the  entire  Strategy  Division.  Accordingly,  the  analyses  carried  under 
the  OEAVT  program  focused  more  exclusively  on  the  operational  and  behavioral  activities 
carried  out  by  the  Operational  Assessment  Team  in  support  of  Operational  Assessment.  The 
analyses  we  conducted  were  assessment-wide,  that  is;  information  bearing  on  all  aspects  of 
assessment,  from  pre-execution  assessment  planning  through  post-execution  archival  activities, 
was  collected  and  analyzed.  As  part  of  our  systems  analysis,  the  OEAVT  team  then  selected  a 
subset  of  this  characterization  of  assessment  for  further  development.  The  AECV  was  the 
centerpiece  of  that  development. 

The  analyses  carried  out  by  the  OEAVT  team  relied  on  3  primary  data  sources.  The  first  was 
existing  documentation  about  operational  and  cognitive  work  aspects  of  Operational  Assessment. 
The  team  reviewed  material  in  this  category  including  instructional  materials,  AFTTP 
materials,  operational  manuals,  publications  and  papers  presented  at  conferences  and  scholarly 
papers  addressing  the  subject  of  assessment  (theses,  dissertations  and  so  on).  A  second 
information  source  included  attendance  at  exercises  in  which  assessment  played  a  prominent  role 
and  at  workshops  and  other  meetings  involving  the  strategy  planning  and  assessment 
communities  within  the  Air  Force.  At  these  events  we  had  a  chance  to  interview  operational 
personnel,  observe  operational  and  work  processes  relevant  to  assessment,  and  engage  in 
informal  discussions  of  various  aspects  of  assessment.  The  third  information  source  consisted  of 
detailed  and  ongoing  interactions  with  our  own  SMEs  and  other  expert  members  of  the  Air  Force 
planning  and  assessment  community.  All  of  our  activities  with  all  of  these  information  sources 
were  focused  on  characterizing  the  process  of  Operational  Assessment  from  three  points  of  view: 
an  operational  view,  a  work  view  and  a  behavioral  view.  These  three  views,  captured  in  both  the 
concept  maps  and  the  subsequent  system  model,  are  crucial  and  equally  important  facets  of  a 
successful  system  of  visualization  support.  As  shown  in  Appendix  A,  the  constituents  of  these 
views  included  operational  processes  and  activities,  requirements,  preconditions  and 
contingencies,  products  and  data  (in  the  case  of  operational  views);  systems  and  interfaces,  task 
hierarchies,  organizational  considerations,  and  artifacts  (in  the  case  of  work  views);  and 
processes  and  cognitive  work  elements,  information,  critical  decisions,  goals  and  strategies  (in 
the  case  of  behavioral  views). 

This  information  was  used  by  the  OEAVT  team  to  carry  out  system  analysis  following  the  work 
domain  and  cognitive  engineering  analyses  and  to  develop  the  system  model  for  assessment  that 
followed  the  initial  system  analysis.  It  was  during  this  initial  system  engineering  analysis  that 
the  team  defined  the  boundaries  of  the  system  for  subsequent  development.  That  is,  the  initial 
cognitive  engineering  analysis  defined  assessment.  The  subsequent  system  engineering  analysis 
defined  what  aspects  of  assessment  the  system  would  support.  When  the  boundaries  of  the 
OEAVT  system  had  been  identified,  the  team  then  developed  the  system  model.  The  portions  of 
the  concept  maps  and  critical  decision  analyses  that  were  relevant  to  the  in-scope  system  to  be 
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developed  served  as  the  baseline  documents  for  system  model  definition.  System  model 
development  was  led  by  the  project  system  engineer.  However,  the  model  itself  was  developed 
by  a  team  consisting  of  the  lead  system  engineer,  analysts  that  had  conducted  the  cognitive 
engineering  analysis,  and  the  project  SMEs.  This  ensured  that  the  system  model  contained  the 
information  needed  to  generate  valid  requirements  while  also  containing  important  information 
from  the  work  and  cognitive  analyses.  This  step  was  critical,  as  system  model  development 
often  is  done  solely  as  an  engineering  analysis.  This  can  result  in  a  failure  to  include  crucial 
cognitive  and  work  considerations  in  the  system  requirements.  The  system  model  developed  for 
the  overall  OEAVT  system  is  shown  in  Appendix  B. 

The  system  model  was  used  to  generate  overall  system  requirements.  These  are  shown  in 
Appendix  C.  The  OEAVT  team  used  the  system  requirements  to  identify  major  areas  of 
capability.  These  areas  were  aggregated  into  the  subsystems  of  the  visualization  tool.  In 
general,  our  analysis  indicated  that  assessment  consisted  of  four  major  activities:  Planning 
activities,  the  actual  process  of  assessment,  indicator  and  work  management,  and  post  assessment 
activities.  We  chose  to  base  development  on  assessment  processes  and  indicator/work 
management  based  on  our  analyses  and  interviews  with  operational  assessors.  In  the  next  section 
we  briefly  summarize  its  major  elements  and  features  as  currently  configured. 

11 .1 .3  Current  Status  and  Capability 


The  AECV  supports  effects  assessment  by  permitting  the  assessor  to  contrast  current  air 
campaign  status  with  planned  progress,  scaled  simultaneously  against  effort  and  effectiveness  in 
a  defined  value  space.  These  contrasts  can  be  applied  to  elements  at  all  levels  of  an  effects-based 
plan.  The  AECV  also  supports  management  and  evaluation  of  indicators  and  evidence  by 
providing  configuration  access  to  this  information  at  any  point  during  the  assessment  process. 
Assessors  can  define  new  indicators,  adjust  the  confidence  associated  with  indicators,  and  view 
indicator  data  sources  and  historical  information  for  any  selected  plan  element. 

The  AECV  is  organized  into  three  primary  work  areas,  as  shown  in  Figure  15.  The  progress 
visualization  pane  contains  the  primary  information  upon  which  ongoing  Operational 
Assessment  is  based.  This  pane  contains  information  about  planned  and  actual  progress  of  the 
air  campaign,  presented  as  both  a  summary  snapshot  and  dynamic  trend  information.  The 
navigation/indicator  pane  contains  controls  that  allow  assessors  to  navigate  and  configure  the 
AECV  and  to  explore  and  manage  indicator  information  for  the  plan  elements  shown  in  the 
progress  visualization  pane.  The  configuration  pane,  shown  at  the  bottom  of  Figure  15,  allows 
assessors  to  develop  and  edit  indicator  specifications,  add  new  indicators,  edit  indicator  text  and 
enter  justifications  for  assessment  decisions.  Each  pane  will  now  be  discussed  in  more  detail. 
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Figure  15:  The  AECV  Supports  Assessment  Through  the  Use  of  Three  Panes 


The  AECV  progress  visualization  pane  presents  two  alternative  views  for  assessment  support:  a 
Summary  View  and  a  Trend  View.  In  the  Summary  View,  multiple  effeets,  rendered  as 
operational  objeetives,  are  plotted  on  a  single  AECV  gradient  representing  the  selected  day  (as 
selected  by  the  Day  Controller).  A  summary  view  gradient  is  shown  in  Figure  16. 
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Figure  16:  AECV  Summary  View 


45 


In  the  Trend  View,  multiple  gradients  are  rendered  with  trend  lines  plotted  to  represent  both 
assessed  effeetiveness  and  planned  effeetiveness.  Figure  17  shows  a  trend  view  example. 
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Figure  17:  AECV  Trend  View 

Each  panel  gradient  of  the  trend  view  is  a  compressed  version  of  the  gradient  shown  in  the 
summary  view,  compressed  to  the  unit  of  time  (hour,  day,  month,  etc.)  chosen  by  the  assessor. 
Panels  contain  the  same  two  dimensions  of  effort  and  effectiveness  that  are  contained  in  the 
summary  view  as  well  as  the  three  adjustable  areas  of  certainty  as  shown  in  the  summary  view. 
Each  panel  can  be  adjusted  in  width.  Two  points  are  contained  in  each  panel.  One  point 
represents  the  planned  effort/effectiveness  for  the  time  unit  chosen.  This  point  will  always  be 
placed  to  the  far  right  edge  of  each  panel,  as  one  always  expects  to  expend  all  effort  that  is 
planned.  The  location  of  the  point  vertically  will  be  a  function  of  expected  effectiveness  for  the 
time  period  in  question.  The  second  point  represents  the  actual  effort/effectiveness  for  the  time 
period  of  interest.  The  placement  of  this  point  will  vary  in  both  the  horizontal  (effort)  and 
vertical  (effectiveness)  dimensions  as  a  function  of  actual  air  campaign  results.  Both  points  are 
color-coded  to  indicate  the  confidence  associated  with  their  placement.  Obviously,  the  points 
representing  planned  effort/effectiveness  will  always  be  coded  green.  The  actual  point  can  vary 
from  red,  through  yellow,  to  green.  The  distance  between  the  points  indicates  “deviation  from 
plan.”  Points  are  connected  with  lines  across  time  gradients,  thereby  showing  a  trending  of 
effort  and  effectiveness  across  the  time  increments  chosen  by  the  assessor. 

The  AECV  addresses  several  specific  areas  of  assessment  within  these  two  broad  categories. 

The  system  allows  assessors  to  load  a  range  of  OA  plans  developed  in  other  applications,  such  as 
MS  Excel,  for  use  in  assessment.  This  allows  the  assessment  team  maximum  flexibility  in 
developing  assessment  plans  according  to  the  methodologies  unique  to  their  operating 
environments. 


46 


Once  an  OA  plan  is  loaded  assessors  can  configure  the  AECV  progress  visualization  pane  by 
setting  the  values  of  the  assessment  gradient,  defining  the  number  of  panels  in  the  trend 
assessment  display,  defining  the  units  of  assessment  (months,  days,  hours).  A  summary  gradient 
or  a  set  of  trend  panels  for  a  plan  element  of  the  assessor’s  choice  can  be  displayed. 

Contrasting  actual  progress  at  specific  points  during  the  air  campaign  to  planned  progress  plotted 
out  at  the  beginning  of  the  campaign  is  one  of  the  central  responsibilities  of  the  OA  team.  It  is 
crucial  that  assessors  be  able  to  see  this  progress  (or  lack  of  progress)  (1)  at  a  glance  (2)  with 
some  indication  of  the  confidence  that  they  can  place  in  the  displayed  progress,  (3)  with  some 
indication  of  the  direction  in  which  progress  is  trending,  and  (4)  with  information  that  provides 
insight  into  how  best  to  troubleshoot  problematic  areas.  Members  of  the  OA  team  contrast  actual 
to  planned  progress  toward  effects  by  comparing  trend  lines  representing  these  two  indices  on 
the  work  pane  portion  of  the  AECV.  The  trend  lines  are  overlaid  on  a  gradient  describing  a 
value  space  in  dimensions  of  effort  and  effectiveness.  This  four-valued  space  is  rendered  as  a 
continuous  gradient  extending  from  green  (low  effort  -  high  effectiveness),  through  yellow,  to 
red  (high  effort  -  low  effectiveness).  Each  trend-line  anchor  is  rendered  in  one  of  three  colors  to 
indicate  the  confidence  in  the  assessment  i.e.,  in  the  placement  of  the  anchor  on  each  unit 
gradient.  These  colors  are  red  (low  confidence),  yellow  (guarded  confidence)  and  green  (high 
confidence).  The  placement  of  these  trend  lines,  anchored  to  each  unit  gradient,  across  an 
assessor-selected  period  of  time  provides  the  OA  team  with  at-a-glance  indications  of  progress 
toward  effects,  with  confidence  estimates  attached  to  these  contrasts.  The  plan  element 
displayed  provides  assessors  with  the  information  needed  to  quickly  target  troubleshooting 
activities  if  progress  seems  to  be  non-existent  or  trending  downward. 

Assessments  of  progress  toward  effects  have  little  meaning  without  accompanying  estimates  of 
confidence.  The  OEAVT  system  provides  two  types  of  confidence  estimates.  In  the  first  type, 
confidence  in  the  progress  toward  effects  is  shown  as  variations  in  the  colors  of  trend  line 
anchors  in  the  progress  visualization  pane.  This  was  summarized  above.  The  second  type  of 
confidence  estimate  applies  to  indicator  performance  and  is  shown  in  the  navigation/indicator 
panel.  The  confidence  in  each  indicator  will  be  shown  as  a  small  block  under  the  indicator  text 
containing  one  of  three  letter  codes  for  low  (L),  moderate  (M)  and  high  (H)  confidence.  These 
blocks  also  are  color  coded,  as  shown  in  Figure  18.  The  confidence  associated  with  indicators 
can  be  set  manually  in  the  configuration  pane.  The  top  portion  of  the  indicator  summary 
contains  a  configurable  display  that  provides  at-a-glance  updates  of  the  performance  of  each 
indicator  for  a  plan  element  of  interest.  This  display  is  both  location  and  color-coded.  Each 
individual  indicator  is  shown  in  separate  panels  under  the  configurable  display.  Each  panel 
contains  the  indicator  text,  the  plan  element  to  which  the  indicator  “belongs,”  confidence 
estimates  for  the  indicator,  a  configurable  diagram  summarizing  the  performance  of  data  used  to 
index  the  indicator,  and  indicator  values  as  defined  in  the  OA  plan. 
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It  also  is  important  for  the  OA  team  to  be  able  to  prediet  the  effectiveness  associated  with  plan 
elements.  This  is  accomplished  in  the  progress  visualization  pane  by  extending  the  prediction 
trend  line  across  gradients  associated  with  future  assessment  units.  For  example,  as  shown  in 
Figure  18,  a  predicted  plan  line  might  extend  3  days  into  the  future.  The  anchor  points  of  these 
expected  trend  lines  will  always  be  placed  at  the  maximum  of  the  effort  scale,  since  one  expects 
that  all  planned  effort  will  be  expended.  Anchor  placement  in  the  effectiveness  dimension  will 
be  a  function  of  the  predicted  effectiveness  for  the  unit  in  question. 


Figure  18:  Indicator  Confidence  Values  are  Shown  as  Coded  Blocks  Under 

Indicator  Text 


The  management  of  indicators  is  of  almost  equal  importance  to  the  actual  assessments.  An 
Operational  Assessment  is  invalid  if  it  is  based  on  the  “wrong”  indicators  or  if  the  “right” 
indicators  are  not  configured  in  a  way  that  they  provide  a  meaningful  context  for  assessments. 
Effective  indicator  management,  then,  consists  of  two  co-equal  parts:  defining  a  diagnostically 
coherent  set  of  indicators  for  the  plan  elements  defined  by  the  plans  team  and  dynamically 
configuring  the  indicator  set  so  that  assessments  are  interpreted  in  ways  that  can  provide  near- 
optimal  information  to  the  plans  team  and  JFACC.  The  OEAVT  system  supports  these  two 
goals  through  the  configuration  panel.  The  configuration  panel  is  used  to  define  indicators  for 
each  plan  element;  assign  values  and  weights,  or  edit  these,  for  each  indicator;  set  confidence 
zones  associated  with  air  campaign  results;  edit  indicator  data  sources;  and  adjust  plan  element 
value  status.  The  design  of  these  AECV  indicator  management  components  benefited  greatly 
from  analyses  and  concept  development  carried  out  by  the  ManTech  Cognitive  Systems 
Engineering  Center  (CSEC),  under  sub-contract  to  SAIC. 
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Figure  19  shows  the  configuration  panel  for  specifying  indicators  and  their  weights.  As 
indicators  are  fed  into  the  AECV  from  the  value  model  they  are  added  to  the  indicator  panel 
shown  in  Figure  19.  Their  weights  then  can  be  dynamically  specified  as  the  air  campaign 
evolves  by  moving  the  pointers  on  the  adjustment  bar  shown  at  the  right  of  Figure  19.  Weights 
can  be  locked  for  particular  indicators  by  clicking  the  lock  symbol  to  the  right  of  each  indicator 
text. 


Dynamically  specify  indicator  weights  in  value  model 


Figure  19:  Panel  Used  to  Edit  indicators  and  Weights 

Confidence  zones  for  each  plan  element  can  be  set  with  the  confidence  zone  editor  shown  in 
Figure  20.  Based  on  information  associated  with  each  indicator  in  the  value  model,  confidence 
zones  will  be  set  that  reflect  assessor  judgments  of  the  points  at  which  results  indexed  by  that 
indicator  pass  from  unfavorable,  through  cautionary,  to  favorable  outcomes.  Assessors  can  edit 
these  values  dynamically  simply  by  moving  the  boundary  pointers  to  a  new,  desired  location  on 
the  confidence  bar. 


Figure  20:  Confidence  Zone  Editor 

Indicator  data  sources  also  can  be  edited,  as  shown  by  Figure  21.  For  each  indicator,  shown  at 
the  top  of  the  display  against  a  yellow  background,  there  will  be  several  data  sources  that 
collectively  contribute  to  the  evaluation  of  the  indicator.  These  data  sources  are  updated  as  new 
results  become  available,  allowing  assessors  to  inspect  them  dynamically  as  the  air  campaign 
unfolds.  The  source  of  each  report  is  shown  to  the  left  of  the  report  text,  and  will  include  each  of 
the  normal  intelligence  sources.  Confidence  is  shown  next  to  each  source.  The  time  of  the 
report  also  is  shown  to  the  right  of  the  report  itself  Using  this  panel,  assessors  can  page  through 
the  sources  for  each  indicator  by  selecting  a  desired  indicator  with  the  drop-down  indicator 
selector.  Indicator  weight,  value  and  confidence  can  each  be  edited  through  the  use  of  the  panel 
to  the  right  of  the  indicator  text. 
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Figure  21 :  Indicator  Data  Sources 


Assessors  can  edit  plan  elements  dynamically  by  using  the  plan  element  value  status  panel, 
shown  in  Figure  22.  This  panel  is  both  a  status  panel  and  an  editing  panel.  The  status  portion  of 
the  panel  is  shown  on  the  left  side  of  Figure  22.  This  area  displays  a  daily  summary  of  the  plan 
element  of  interest  for  both  the  effectiveness  and  effort  dimensions.  Planned  values  in  each  of 
these  dimensions  are  shown  as  blue  histogram  elements,  with  the  plan  value  printed  inside  the 
histogram.  Actual  values  are  shown  immediately  below,  again  as  histograms  with  values  printed 
inside.  This  allows  both  an  at-a-glance  comparative  evaluation  of  progress  toward 
accomplishment  of  the  plan  element  as  well  as  a  more  analytical  evaluation  (if  so  desired)  based 
on  the  digital  values.  This  is  in  keeping  with  a  general  principle  of  visualization  design:  Support 
quick,  effortless  (analogical)  evaluations  of  progress  through  at-a-glance  visual  “thinking”  along 
with  more  effortful,  analytical  (digital)  inspection  or  exploration,  if  needed  or  desired.  The  right 
side  of  the  plan  element  value  status  panel  allows  assessors  to  edit  plan  element  weights  and 
confidence  so  as  to  keep  them  aligned  with  the  evolution  of  the  air  campaign.  When  this 
information  is  changed,  assessors  will  typically  be  required  to  provide  some  justification  for  the 
changes.  This  is  supported  with  the  addition  of  a  justification  block  under  the  editing  panel. 


Figure  22:  Plan  Element  Value  Status 
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When  results  deviate  from  the  plan,  it  will  be  necessary  for  the  OA  team  to  “troubleshoot”  the  air 
campaign.  All  three  panes  in  the  AECV  are  used  to  successfully  carry  out  this  troubleshooting. 
The  progress  visualization  pane  is  used  to  detect  the  presence  of  a  deviation  between  planned 
and  actual  progress.  Additionally,  assessors  use  this  pane  to  determine  the  general  area  of  the 
plan  in  which  the  problem  exists.  That  is,  is  the  deviation  from  plan  at  the  operational  objective 
level,  the  tactical  objective  level  or  the  tactical  task  level?  The  progress  visualization  pane 
supports  this  exploration  by  allowing  the  assessor  to  search  hierarchically  through  plan  elements, 
inspecting  deviations  from  each  element  in  the  hierarchy.  This  hierarchical  search  is  shown  in 
Figure  23. 


Figure  23:  Hierarchical  Search  Through  Pian  Elements 


The  top-level  summary  display  shows  an  operational  objective  in  the  red  area,  indicating  lack  of 
progress  toward  effects.  Its  green  code  indicates  high  confidence  in  the  lack  of  progress.  The 
assessor  then  drills  down  to  one  of  the  tactical  objectives  (TO)  in  an  effort  to  locate  the  reasons 
for  the  lack  of  progress.  This  is  shown  in  the  middle  display.  Here  we  can  see  that  there  is  a 
discrepancy  between  planned  and  actual  progress  in  the  third  panel.  The  assessor  then  drills 
down  further  to  some  of  the  tactical  tasks  for  that  TO,  finding  the  problem  with  tactical  task 
CL  IB.  Specific  corrective  action  then  can  be  recommended.  Alternatively,  the  assessor  might 
make  adjustments  to  the  value  model,  the  indicators  or  the  data  upon  which  the  indicators  are 
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based  in  order  to  tune  the  dynamic  assessment  plan.  By  searching  in  increasing  detail  from  top- 
level  summary  to  individual  operational  objectives,  and  then  selectively  down  through  tactical 
objectives  to  tactical  tasks,  assessors  can  pinpoint  the  sources  of  plan  deviations.  They  then  can 
inspect  individual  and  groups  of  indicators  associated  with  the  focal  plan  element  to  determine 
the  “causes”  of  plan  deviations. 

A  pervasive  aspect  of  Operational  Assessment  is  the  need  to  manage  uncertainty.  The 
uncertainty  inherent  in  Operational  Assessment  arises  from  several  sources  including 
measurement  uncertainty  associated  with  selection  and  implementation  of  indicators,  the 
uncertainty  associated  with  the  accuracy  and  timeliness  of  results  during  ongoing  air  operations, 
uncertainty  arising  from  the  combining  of  individual  assessments  into  aggregate  assessments 
(often  referred  to  as  assessment  roll-ups),  and  uncertainty  arising  from  indicator  specifications 
(that  is,  is  this  indicator  specified  in  a  way  that  is  meaningful  to  the  needs  of  the  assessment?). 
The  OEAVT  system  implements  several  means  of  addressing  uncertainty  in  Operational 
Assessment.  In  terms  of  infrastructure  the  system  is  designed  to  provide  reporting  of  strike 
results  as  they  become  available,  thereby  minimizing  uncertainty  due  to  timeliness.  Uncertainty 
due  to  indicator  measurement  accuracy  is  addressed  by  allowing  assessors  to  define  and 
dynamically  reset  the  confidence  associated  with  each  indicator.  The  uncertainty  that  can  arise 
from  initial  definition  of  indicators,  carried  out  during  OA  planning,  can  be  addressed  through 
the  use  of  an  archival  indicator  library  that  contains  features  such  as  intelligent  search,  system 
suggestions  regarding  indicators  that  have  been  useful  in  past  operations  of  a  similar  nature  and 
so  on.  Although  this  library  was  not  implemented  in  the  current  version  of  OEAVT,  design 
concepts  were  developed  and  have  been  included  in  archival  documentation  contained  in  the 
appendices.  Uncertainty  also  arises  in  connection  with  the  type  of  operation  being  undertaken 
and  with  the  methodology  in  use  for  assessment.  Some  types  of  operations  (e.g.,  traditional 
kinetic  operations)  will  tend  to  be  “less  uncertain”  than  other  types  (e.g.,  peacekeeping  or 
humanitarian  operations).  In  the  former  example  both  the  indicators  and  the  assessment 
methodologies  tend  to  be  more  familiar,  more  “trustworthy,”  more  easily  understood  and 
interpreted  by  assessors.  This  is  less  so  in  the  latter  examples.  Thus,  an  effective  assessment 
support  system  should  allow  assessors  to  express  this  “contextual  uncertainty.”  The  OEAVT 
system  accomplished  this  through  the  use  of  a  configurable  gradient,  allowing  assessors  to  define 
regions  of  uncertainty  as  well  as  positive  and  negative  certainty  for  each  of  the  plan  elements 
contributing  to  overall  Operational  Assessment.  This  gradient  allows  assessors  to,  in  effect; 
visualize  their  impression  of  the  risk  associated  with  the  operation  under  assessment.  The  last 
source  of  uncertainty  that  the  OEAVT  system  addressed  was  that  of  effort/effectiveness  ratios. 
Both  of  these  dimensions  are  important  to  a  complete  assessment.  It  is  not,  however,  acceptable 
to  consider  them  in  isolation.  Rather,  individual  plan  element  assessments  should  allow  at-a- 
glance  understanding  of  deviations  in  both  dimensions.  The  OEAVT  system  accomplished  this 
requirement  by  presenting  trend  lines  varying  in  both  effort  and  effectiveness  dimensions 
simultaneously. 

In  summary,  the  AECV  component  of  the  OEAVT  provides  the  capability  to  create  the 
visualizations  which  aid  the  assessor  in  determining  if  the  desired  operational  effects  are  being 
achieved  during  the  execution  of  the  battle  plan.  The  OEAVT  system  software,  (allows  the 
assessor)  to  assess  the  level  to  which  desired  operational  objectives  are  being  achieved  by 
determining  the  extent  to  which  planned  actions  are  achieving  planned/desired  effects.  The 
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software  system  provides  the  Assessor  with  at-a-glance  insight  into: 


•  Air  operations  progress,  sources  and  causes  of  plan  deviations 

•  Analysis  capability  to  enable  effective  development  of  recommendations  to  Air 
Component  Commanders 

•  Assess  progress  toward  operational  objectives  by  reference  to  the  overall  plan  element 

•  Assess  progress  toward  tactical  objectives 

•  Actual  and  expected  progress,  and  finally, 

•  Identify  new  opportunities  for  actions  or  effects 

While  AECV  and  the  OEAVT  system  software  provides  discrete  quantitative  assessment  data, 
its  strength  lies  in  how  it  draws  the  assessor’s  attention  to  areas  in  the  strategic  plan  that  are 
relevant  in  terms  of  assessment. 

1 1 .2  Geo-spatial  Effects  Visualization  (GEV) 


11 .2.1  Purpose 


The  Geo-spatial  Effects  Visualization  (GEV)  was  proposed  to  address  the  shortcoming  of  bar 
graphs  and  stoplight  charts  in  portraying  assessments  in  geo-spatial  context.  The  GEV  sought  to 
preserve  the  simplicity  of  stoplight  assessment  colors  assigned  to  indicators  placed  in  geo-spatial 
context.  This  map  display  provided  understanding  not  only  of  the  assessment  results,  but  the 
adequacy  of  the  assessment  model  in  geo-spatial  terms:  Were  indicators  being  sampled  in 
critical  corridors?  Did  assessed  effects,  when  plotted  spatially,  suggest  new  avenues  to  strike  at 
an  adversaries  center-of-gravity?  The  GEV  display  was  intended  to  provide  at-a-glance 
presentation  of  information  to  address  such  questions. 

1 1 .2.2  GEV  Development 


An  initial  whitepaper  was  written  and  presented  to  71 1  HPW/RHCP  that  detailed  the  reasons 
why  Geo-spatial  Effect  Visualization  is  necessary  in  effect  assessment,  listed  the  data  sources 
required  to  implement  the  capability,  and  suggested  approaches  for  implementation.  (Geo¬ 
spatial  Effects  Visualization  (GEV)  -  A  White  Paper  For  Air  Force  Research  Laboratories  / 
Human  Effectiveness). 

An  internal  study  was  conducted  on  the  human  perception  of  symbols  on  maps,  with  an  emphasis 
of  understanding  overlays  on  depictions  of  terrain,  lines  of  communication,  and  political 
boundaries.  The  guidelines  resulting  from  this  effort  were  documented  in,  OEA  VT  Concept 
Design  Principles:  Map  Displays,  27  January  2006,  which  is  attached  as  Appendix  D. 
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The  application  of  some  of  these  design  principles  can  be  seen  in  an  Adobe  Photoshop®  mockup 
of  a  GEV  display,  depicted  in  Figure  24.  The  following  display  characteristics  are  used: 

Stoplight  colors  based  on  the  CIE-LAB  color  space  designed  to  allow  legibility  of 
overlay  text. 

Shaded  terrain  relief  and  unsaturated  colors  in  the  base  map  graphic  to  enhance 
legibility  of  overlays. 

A  later  GEV  concept,  using  transparency  to  indicate  the  “weight”  of  an  indicator,  thus 
showing  the  importance  of  the  red  indicator  relative  to  the  other  indicators  by  the 
intensity  of  its  surrounding  “halo”. 


Figure  24:  GEV  Photoshop  mockup 


A  Trade  Study  was  conducted  to  ascertain  the  functional  capabilities  of  several  candidate 
mapping  applications  against  notional  functional  requirements  (Appendix  D).  Candidates 
reviewed  included  OpenMap,  ArcGIS,  OSSim,  Google  Earth  Basic  and  Pro,  Google  Maps, 
JView,  and  uDig.  From  this  trade  study,  OpenMap  was  selected  as  the  mapping  tool  to  be 
utilized  in  initial  GEV  design  and  prototyping.  OpenMap’s  main  benefits  were:  It  is  open 
source,  and  it  has  a  full  API,  fully  configurable  from  GUI  layout  to  data  handling  for  each  layer. 
It’s  only  real  detraction  was  that  it  is  solely  a  2D  mapping  library.  But  there  were  no  plans  to 
add  any  3D  capabilities  in  the  first  design. 

An  initial  design  was  created  around  the  OpenMap  library.  OpenMap  is  layer  based,  and  so  the 
GEV  was  designed  around  a  layer/overlay  architecture.  Initial  layers  were  a  Base  Layer 
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containing  by  use  selection  either  rendered  DTED  data  or  CADRG  imagery.  Overlaid  on  the 
base  layer  would  be  one  or  more  PMESSI  layers — Politieal,  Military,  Economic,  Social, 
Infrastructure,  and  Information.  The  initial  design  would  focus  on  basics  like  city  and  state 
names,  roads,  utilities.  The  top  overlays  would  eonsist  of  the  effeet  assessments. 

A  GUI  Conceptual  design  was  ereated,  as  depicted  in  Figure  25  below. 


GEV  GUI  Concept 


Figure  25:  GEV  GUI  Concept 

From  that,  OpenMap  was  used  to  ereate  a  set  of  design  mockups.  (Appendix  D).  The  core  of  the 
design  was  the  visualization  of  effeets  by  an  area  of  effect  that  would  be  eolored  per  stoplight 
status  (Green,  Yellow,  Red),  with  a  solid  outline.  The  area  for  a  given  element  would  be 
generated  from  aggregating  child  plan  element  locations,  or  at  the  lowest  level,  target  locations. 
Plan  element  type  would  be  shown  through  the  use  of  icons.  The  icons  are  also  colored  per 
stoplight  status,  and  the  area  around  the  icon  would  be  filled  with  a  level  of  transparency 
correlating  to  the  weight  of  that  plan  element’s  assessment.  Figure  26  depicts  the  basic  design 
moekup  in  sehematie  form,  showing  the  map  area,  assessment  summary  symbol,  geographie  and 
non-geographie  assessment  symbols,  and  a  basie  toolbar: 
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Figure  26:  GEV  schematic  design  mockup  rendered  in  OpenMap 


1 1 .2.3  Current  Status  and  Future  Development 


Due  to  budgetary  constraints,  further  development  of  geo-spatial  effects  was  put  on  hold  in  June, 
2007.  The  design  was  and  remaining  issues  were  documented  in  OEAVT  Geo-spatial  Effects 
Visualization  (GEV)  Software  Design  Notes,  July  2,  2007,  previously  delivered  to 
711  HPW/RHCP. 

11.3  TBONE-IWPC  Indicator  Interface  (TI3) 


11.3.1  Purpose 


The  TBONE-IWPC  Indicator  Interface  (TI3)  was  conceived  as  a  proof-of-concept  machine-to- 
machine  integration  between  IWPC  and  TBONE.  Its  intent  was  to  capture  information  residing 
in  the  TBONE  strike  results  database  and  make  it  available  to  the  IWPC  assessment  component 
(the  Enhanced  Causal  Analysis  Tool  (eCAT)  module),  thereby  making  visible  to  the  OAT 
information  bearing  on  Operational  Assessment.  Under  this  vision,  TI3  was  officially  designated 
as  the  “Indicator  Support  System”  within  the  IWPC  development  community. 

From  its  conception,  the  intent  was  for  TI3  to  be  a  simple,  low-impact  augmentation  to  eCAT. 

To  realize  this  intent,  and  in  keeping  with  program  desires  and  engineering  constraints 
articulated  by  the  primary  IWPC  integrator  (General  Dynamics),  SAIC  isolated  the  TI3 
integration  code  into  a  separate  module  accessible  from  eCAT.  This  kept  any  required  changes 
to  eCAT  at  a  minimum. 
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1 1 .3.2  Development 


The  genesis  for  TI3  arose  during  a  Strategy  and  Assessment  (S&A)  Requirements  Sub-Working 
Group  (RWSG)  Technical  Exchange  Meeting  (TIM)  held  in  March,  2005.  This  TIM  was  held  to 
investigate  data  requirements  for  TBONE.  A  consideration  of  Operational  Assessment  during 
this  meeting  indicated  that  a  need  existed  to  make  tactical  strike  results  available  to  IWPC,  the 
Operational  Assessment  system  of  record.  The  OEAVT  team  was  tasked  to  develop  a  solution 
that  would  accomplish  this  integration. 

Upon  approval  by  the  IWPC  integrator  of  the  TI3  system  engineering  plan,  SAIC  began 
prototyping  a  concept  for  enabling  search  and  display  of  TBONE  strike  results  and  other 
assessment-relevant  data  in  eCAT.  The  first  milestone  was  a  simple  proof-of-concept  search 
capability  that  would  allow  analysts  to  access  TBONE  data  services,  located  on  TBMCS 
DevNet,  in  order  to  acquire  tactical  information  to  support  Operational  Assessment. 

The  centerpiece  of  this  capability  was  the  TBONE  Data  Surf  Board  (TBDSB).  The  key  to  the 
surfboard  is  the  TBONE  MetaData,  which  would  allow  us  to  pre-populate  a  set  of  drop-down 
boxes  with  data  that  were  implicitly  accurate  for  that  instance  of  TBONE  (since  it  was  based  on 
the  meta-data).  For  the  first  iteration  of  the  TBDSB,  we  derived  our  metadata  using  Java 
introspection  of  the  TBONE  DataObjectSet  class.  This  helper  application  allows  developers  to 
construct  and  execute  TBONE  ad  hoc  queries  without  having  to  re-write  the  program.  The  team 
developed  a  TBDSB  GUI  and  successfully  loaded  a  hand-built  TAS  Object  (the  base  class 
WebTAS  object  in  the  TBONE  data  model).  In  the  mean  time,  we  experimented  with  a  rote 
TBONE  counter  (TBDC)  that  did  nothing  more  than  traverse  the  TBONE  meta-data  and  return 
counts  of  all  the  object  instances  in  TBONE,  thereby  creating  a  phase  1  working  version  of  the 
TBDSB  that  allowed  simple  queries  to  be  made. 

In  parallel  with  phase  1  TBDSB  development,  SAIC  proceeded  with  initial  modifications  to 
eCAT.  In  keeping  with  the  engineering  constraint  of  "minimal  changes  to  eCAT,"  SAIC 
implemented  a  single  change  to  the  eCAT  GUI  in  the  form  of  an  additional  button  (circled  in 
Figure  27). 
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Figure  27:  TI3  Modification  to  eCAT  GUI 


The  second  version  of  the  TBDSB  made  direct  use  of  meta-data  available  in  TBONE  rather  than 
using  Java  introspection.  This  reliance  on  TBONE  meta-data  greatly  improved  the  stability  of 
the  surfboard.  Assessment  data  within  the  TBONE  data  model  were  associated  with  TBONE 
strike  results,  as  shown  in  Figure  28.  In  order  to  access  this  information,  assessors  entered 
queries  based  on  elements  of  the  effects-based  plan  and  indicators  associated  with  those  plan 
elements.  This  information  was  then  drawn  from  the  TBONE  tactical  strike  results  database. 
The  results  were  provided  to  assessors  through  a  TI3  interface  integrated  into  the  eCAT  module 
oflWPC. 
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Figure  28:  TBONE  Assessment  Data  Chain 


Concurrent  with  the  second  phase  of  TBDSB  development,  SAIC  developed  a  pop-up  window 
for  integration  into  IWPC  that  would  have  the  proper  IWPC  look-and-feel  and  interact  correctly 
with  the  rest  of  the  IWPC  windows  (i.e.  close  when  the  parent  window  closed,  etc.).  The 
resulting  concept  of  operations  for  the  TI3  is  represented  in  Figure  29. 


Operator 


Figure  29:  TI3  ConOps 
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1 1 .3.3  Current  Status 


The  TI3  is  invoked  from  inside  eCAT  by  elieking  the  "Query  Exeeution"  button  on  the  eCAT 
"Assess  Status"  sereen  (Figure  27).  This  action  will  display  the  TI3  "Indicator  Assessment"  pop¬ 
up  window,  initialized  with  query  parameters  based  on  the  current  Effect  and  Indicator  selected 
in  eCAT  (Figure  30). 


Figure  30:  TI3  Indicator  Assessment  initial  Screen 


The  assessor  clicks  the  "Execute"  button  (middle-left)  to  initiate  the  TBONE  query.  The  "Data 
Navigator"  pane  (center)  is  then  populated  in  real  time  as  assessment  data  are  retrieved  from 
TBONE.  Assessors  are  free  to  peruse  results  while  queries  are  in  progress.  When  a  query  is 
complete,  the  "Status  Window"  (lower-left)  will  indicate  "Query  finished." 

Selecting  a  result  line  in  the  Results  table  will  reveal  the  details  of  that  Strike  Request  in  the  pane 
below  the  table. 

The  Indicator  Assessment  window  can  be  switched  between  two  display  formats.  The  first 
format  is  keyword  search  mode  (Figure  31),  allowing  the  assessor  to  perform  an  automated 
search  for  one  or  more  keywords.  In  this  format,  matching  results  are  highlighted  in  the 
"Results"  area  and  the  corresponding  data  element  is  highlighted  in  the  target  tree  displayed  in 
the  Data  Navigator  panel.  The  Result  table  scrolls  as  needed  to  reveal  the  highlighted  search 
text. 
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Figure  31 :  TI3  Keyword  Search  Format 


The  second  format  is  browse  mode  (Figure  32),  which  maximizes  the  view  of  the  results  while 
hiding  the  parameter  panel,  the  keyword  search  panel,  and  the  status  window.  Assessors  can 
switch  back  to  keyword  search  format  by  clicking  the  "Return"  button  at  the  top-left  of  this 
display. 
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Figure  32:  TI3  Browse  Format 


In  April  of  2007,  the  TBMCS  Program  Office  officially  cancelled  TBMCS  1.1.4,  including  the 
TBONE  development  effort.  This,  in  effect,  also  cancelled  the  TI3  effort.  Thus,  even  though  the 
capability  was  included  in  the  SQT  version  of  IWPC,  the  OEAVT  team  ended  active 
development  of  the  module  at  that  point. 


11.4  Global  Effects  Matrix-Synchronization  (GEM-S)  lO  Assessment  Visualization 
Design  and  Prototype 

11 .4.1  Purpose 


This  initial  concept,  further  described  in  the  following  section,  identified  Influence  Operation 
(IFO)  activities  accomplished  by  multiple  organizations,  and  while  recognizing  the  need  for 
assessment,  did  not  provide  any  visualizations  for  assessments.  The  Joint  Forces  Command 
(JFCOM)  subsequently  funded  an  SRA  International  effort  to  adopt  the  AFRL/RHC  Course  of 
Action  (COA)  Sketch  tool  to  GEM-S  needs,  with  SAIC  contributing  Information  Operations 
(10)  assessment  visualization  concepts. 
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1 1 .4.2  GEM-S  Assessment  Development 


Development  of  the  GEM-S  Assessment  Visualization  Prototype  was  a  short-term  effort  to 
address  a  lack  of  Influence  Operations  (IFO)  assessment  visualizations  to  support  broad-ranging, 
national-level  10  goals  for  the  JIOWC).  This  effort  originated  at  an  Air  Force  Research 
Laboratory  (AFRL)  workshop  where  SAIC’s  Operational  Effect  Assessment  Visualization 
Technology  (OEAVT)  tools  for  visualizing  the  assessment  of  effects-based  air  operations  were 
demonstrated  to  JFCOM  personnel. 

An  initial  elicitation  trip  to  the  GEM-S  customer,  the  Joint  Information  Operations  Warfare 
Center  (JIOWC),  introduced  the  broad  scope  of  IFO  activities,  and  the  need  for  visualizing  and 
assessing  the  various  IFO  activities  associated  with  this  plan.  Classified  topics  were  discussed  in 
the  elicitation,  which  are  not  further  described.  A  spreadsheet-based  visualization  called  the 
GEM-S  had  been  constructed  by  the  JIOWC/J24.  This  initial  implementation  of  GEM-S,  also 
known  as  the  “horse  blanket,”  mapped  national  IFO  objectives  to  a  Primary  Effects  List  (PEL), 
and  in  turn  to  an  activity  performed  by  a  U.S.  Government  (DoD,  State  or  other  agency)  or  non- 
U.S.  government  activity.  The  status  of  the  activity  was  color-coded  to  indicate  active,  planned, 
or  needed.  In  this  manner,  the  color  code  cells  of  the  spreadsheet  provided  a  rapid  view  of  who 
was  doing/planning  what  (at  the  level  of  program  title),  versus  the  objectives  of  the  NIP.  A 
column  labeled  “assessment,”  mapped  to  objectives,  was  blank. 

An  unclassified  surrogate  of  the  “horse  blanket”  is  depicted  in  Figure  33,  and  summarizes  the 
basic  elements  of  the  synchronization  matrix  using  a  fictitious  example  of  introducing  electric 
vehicles  to  the  U.S. 
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Figure  33:  Unclassified  surrogate  for  the  GEM-s  “Horse  Blanket” 


A  second,  unclassified  elicitation  session  was  held  19  December  2007  at  SRA’s  San  Antonio 
facilities,  and  covered  specifically  the  topic  of  IFO  assessment.  SAIC  presented  a  small  set  of 
slides  describing  an  IFO  assessment  concept,  and  received  constructive  feedback  on  IFO 
assessment. 
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The  assessment  visualization  eoneepts  for  IFO  presented  during  the  seeond  elieitation  foeused  on 
eharaeterizing  the  aehieved  effeet  of  an  IFO  aetivity  in  terms  of  pereeption  of  the  delivered 
message  (direetion)  and  penetration  of  the  message  in  the  target  population  (magnitude).  Figure 
34  provides  an  example  graphie  from  that  elieitation.  The  eoneept  of  magnitude  verses  direetion 
was  well  reeeived,  and  SAIC  proeeeded  with  further  development  of  this  eoneept. 
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Figure  34:  Initial  IFO  assessment  concept  graphic  presented  to  JIOWC 


No  attempt  to  plot  achieved  effect  versus  expended  effort  was  made  due  to  the  difficulty  in 
uniformly  quantifying  effort  across  the  activity  domain  of  the  GEM-S.  Unlike  kinetic  air 
operations,  where  effort  is  fairly  uniformly  measured  in  Desired  Mean  Point  of  Impact  (DMPI)  - 
Sortie  -  Equivalents  (DSEs),  no  uniform  measure  of  IFO  effort  is  known  to  exist.  {For  an 
explanation  of  DMPls  and  DSEs  as  applied  to  kinetic  planning  and  assessment,  see  AFTTO  3- 
3.AOC,  1  November  2007  FINAL,  section  3. 3. 3. 3). 

A  final,  informal  elicitation  was  conducted  at  the  JIOWC,  during  the  installation  of  the  GEM- 
S/COA  Sketch  prototype.  Classified  topics  were  discussed  along  with  the  general  nature  of  IFO 
assessment.  A  foreign  media  analyst  and  IFO  assessor  confirmed  the  utility  of  the 
direction/magnitude  assessment  paradigm  and  also  provided  samples  of  typical  raw  input 
provided  to  IFO  assessors  by  commercial  foreign  media  analyst,  highlighting  the  need  to  refer  to 
files  in  addition  to  operator  provided  comments  in  justifying  assessments.  Additional 
discussions  were  conducted  on  10  operations  other  than  IFO.  Jamming  an  adversary’s 
transmitter  is  an  example  of  such  an  operation.  Currently,  the  results  of  such  operations  are 
assessed  on  a  “4-D”  basis  (i.e..  Disrupt,  Deny,  Damage,  Destroy),  which  can  be  thought  of  in 
terms  of  increasing  duration  and  degree  of  effect  on  a  target  system.  This  discussion  prompted 
the  inclusion  of  what  we  will  describe,  for  lack  of  a  general  term,  as  an  “active”  10  display,  as 
opposed  to  IFO,  in  the  prototype  assessment  tool. 
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1 1 .4.3  Current  Status  and  Future  Development 

The  Prototype  Assessment  Visualization  Tool  was  designed  to  support  proof-of-concept 
demonstration  to  users  for  the  purpose  of  refining  and  improving  10  assessment  displays.  As 
such,  it  is  not  designed  as  a  stand-alone  tool,  but  as  a  visualization  “engine,”  which  relies  on  an 
external  system  (COA  Sketch)  for  Operational  Plan  inputs  and  underlying  database.  The 
prototype  assessment  visualization  has  been  integrated  to  utilize  the  current  COA  Sketch 
database  and  API,  and  delivered  to  SRA  International. 

The  original  concept  graphics  were  revised  based  on  internal  peer-review  and  10  SME  review  to 
depict  two  different  basic  assessment  displays:  one  for  IFO,  and  a  second  for  “active”  10.  These 
are  depicted  in  Figure  35  and  Figure  36  respectively.  The  functional  capability  and  visual 
specification  for  the  interface  is  described  in  greater  detail  in  Appendix  E. 

Note  that  no  interface  is  provided  to  manipulate  the  10  plan:  this  is  created  outside  this  tool  in 
COA  Sketch  and  imported  for  display.  Controls  are  available  to  set: 

1)  Assessment  values  associated  with  IFO  direction 

2)  Assessment  values  associated  with  IFO  magnitude 

3)  Assessment  values  for  duration  for  both  desired  and  collateral  indicators  for  active  10. 

4)  Assessment  values  for  percent  reduction  in  capability  for  both  desired  and  collateral 
indicators  for  active  10. 

5)  Confidence  for  all  indicators,  both  IFO  and  active  10. 

6)  Weights  for  all  indicators  and  subordinate  operations. 
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Figure  35:  Basic  IFO  Assessment  Display,  with  a  Direction/magnitude  Grid 
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Figure  36:  Active  iO  Assessment  Display,  with  a  Duration/degree  Capacity 

Reduction  Graph 


The  prototype  assessment  tool  was  evaluated  internally  by  a  mix  of  human  faetor  engineers  and 
by  10  SMEs  eurrently  supporting  the  JIOWC  and  familiar  with  the  10  assessment  process. 
Overall  findings  were  quite  positive,  with  several  general  comments  on  good  features: 

1)  The  direction/magnitude  display  for  IFO  makes  sense  and  is  intuitive  to  understand. 

2)  The  “active”  display  is  also  intuitive  to  understand  and  allows  rapid  characterization  of 
the  success  of  planned  operations. 

3)  Supporting  the  mix/match  of  indicators  and  subordinate  operations  is  flexible  and 
supports  real-world  needs. 

4)  Controls  and  displays  for  assessed  values,  confidence,  weights,  and  age  are  clear  to 
understand  and  easy  to  use. 

5)  After  an  initial  walkthrough,  experienced  10  assessors  found  the  overall  tool  intuitive  to 
use. 

To  support  the  evaluation,  a  fictional  10  scenario  was  constructed,  “Operation  Healing  Voice,” 
and  an  XML  file  was  built  to  provide  an  assessment  snapshot  of  this  operation.  This  allowed 
SMEs  and  engineers  performing  the  evaluation  to  see  how  the  tool  worked  in  context  with  an 
operation.  Several  areas  for  potential  future  improvement  were  identified  in  this  process,  and  are 
discussed  below: 
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1 .  Implement  a  capability  to  allow  assessment  of  performance/effort. 

2.  Provide  help  reference  to  10  TTPs. 

3.  Provide  mechanism  for  operator  to  define  the  indicator  sampling  technique. 

4.  Allow  for  review  of  justification  history. 

5.  Display  multiple  collateral  damage  effects  on  the  active  10  display. 

6.  Improve  the  Operator  Interface. 

7.  Alter  the  depiction  of  active  10  desired  or  collateral  effects  to  ovals. 

8.  Provide  a  detailed  “analyst’s  view’’  toggle  for  the  active  10  display  to  include  point  & 
click  editing  of  effects  bounds. 

9.  Provide  a  stand-alone  capability;  including  local  Op/effect  input  screens  and  database. 

10.  Allow  expected  magnitude  to  be  set  by  indicator,  rather  than  by  effect. 

Implement  a  capability  to  allow  assessment  of  performance/effort.  Current  EBO  doctrine  calls 
for  establishing  both  Measures  of  Performance  (MoPs),  and  Measures  of  Effectiveness  (MoEs). 
The  prototype  tool  focuses  exclusively  on  MoEs  and  their  indicators.  As  discussed  earlier,  this 
design  decision  was  based  on  the  observation  that  performance  (i.e.,  effort),  especially  for  IFO, 
is  difficult  to  measure  in  a  uniform,  repeatable  fashion.  Displays  can  certainly  be  added  to 
evaluate  MoPs,  but  the  fundamental  issue  of  uniform  measurement  must  be  addressed  at  the 
level  of  10  Tactics,  Techniques  and  Procedures  (TTP),  in  order  to  allow  a  meaningful 
comparison  either  within  or  between  10  operations.  This  will  require  significant  SME  input,  and 
subsequent  acceptance  by  the  10  community. 

Provide  help  reference  to  10  TTPs.  Within  the  realm  of  effects,  the  criteria  for  evaluating 
effects  based  on  indicators  follows  TTPs  (business  process  rules),  that  range  from  global  to 
operation  specific.  Recognizing  that  assessors  will  vary  in  background  and  skill,  and  that  the 
assessment  of  IFO  in  particular  is  subjective,  help  screens  or  windows  that  allow  the  operator  to 
review  these  process  rules  in  context  to  setting  assessment  values,  confidences,  and  weights 
would  aid  in  maintaining  consistent  assessments. 

Provide  mechanism  for  operator  to  define  the  indicator  sampling  technique.  Equally  important 
to  assessment  as  selecting  a  measurable  indicator  is  specifying  the  sampling  method  to  be  used. 
No  provision  currently  exists  for  in  COA  Sketch  or  the  assessment  tool  prototype  to  specify 
sampling  method  for  an  indicator.  A  related  issue  is  the  generation  of  an  Assessment 
Information  Request  (AIR)  needed  to  task  collection  assets  under  the  tactical  control  of  other 
commands  to  generate  the  data  needed  to  satisfy  the  sampling  method. 

There  is  also  an  opportunity  to  apply  statistical  theory  as  a  quality  control  measure  to  the 
sampling  method.  By  describing  the  stochastic  nature  of  an  indicator  and  the  sampling  method 
and  frequency  applied,  it  is  possible  to  predict  the  confidence  yielded  by  a  given  sampling 
technique.  This  is  currently  a  standard  procedure  in  opinion  polling,  but  is  not  routinely  applied 
to  other  sample  gathering  techniques  such  as  employing  Intelligence,  Surveillance,  and 
Reconnaissance  Division  (ISRD)  assets.  This  can  be  used  to  determine  if  a  given  indicator  is 
being  sufficiently  sampled,  and  just  as  useful,  when  it  is  being  over-sampled  and  squandering  a 
scarce  collection  asset.  This  metric  does  not  address  whether  an  indicator  is  appropriate  or 
useful,  but  only  that  it  is  being  sufficiently  sampled  to  reach  a  given  level  of  statistical 
confidence. 
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Allow  for  review  of  justification  history.  As  currently  implemented,  the  Assessment  Tool 
requires  that  a  user  enter  text  or  attach  documents  to  justify  eaeh  ehange  in  assessment,  but 
provides  no  ability  to  review  past  justifications.  Such  review  is  essential  to  allow  an  analyst  to 
review  trend  data.  Suggested  funetionality  includes: 

•  Seroll  backwards  in  time  through  assessment  justifieations,  showing  eaeh  justification 
or  attached  file  in  ehronological  order. 

•  Provide  symbols  on  the  assessment  display  to  show  the  approximate  position  of  the 
displayed  justification  on  the  assessment  trend- line. 

•  Display  the  actual  date  of  the  justifieation. 

•  Update  the  displayed  indieator  values  (weight,  assessed  value,  eolor)  eonsistent  with 
the  point  in  time,  and  provide  an  indieation  of  whieh  indicator’s  justification  is  being 
displayed. 

A  eoneept  for  the  history  review  interface  is  shown  in  Figure  37.  Note  that  this  should  be  a 
modal  display:  The  user  may  review  the  history  of  indieator  changes,  but  eannot  adjust  indicator 
values  or  alter  roll-up  assessments  while  in  this  mode. 

Display  multiple  collateral  damage  effects  on  the  active  IQ  display.  Based  on  user  feedbaek, 
multiple  collateral  effeets  may  result  from  10  aetions.  The  most  significant  effects,  not  just  a 
single  effeet,  should  be  viewed  on  the  Aetive  10  display.  To  avoid  rarely  used  eomplexity,  it  is 
suggested  that  the  maximum  number  of  collateral  effeets  displayed  for  a  single  10  task  be  three. 
This  does  not  mean  that  the  user  may  only  ehoose  three  collateral  effects.  Multiple  indicators 
may  be  assigned  and  weighted  to  allow  a  sum  of  minor  effeets  to  be  displayed  as  a  single 
summary  effect.  In  fact,  with  the  display  limit  of  three,  a  useful  convention  may  be  to  display 
the  two  most  signifieant  eollateral  effeets,  and  a  summary  for  remaining  minor  effeets.  The 
presenee  of  multiple  effects  requires  that  a  weighting  scheme  be  implemented  to  summarize 
collateral  effects  for  the  purpose  of  roll-up. 
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Assessment  Explorer 

▼  Operation  Healing  Voice 
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Magnitude  Indicators 
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Justification/file  history  area.  As  you  go  ‘back  in  time't 
cursor  *Q'on  the  history  line  approximates  the  date,  a 
the  indicator  for  the  displayed  justification  is  highlighted. 


The  roll-up  assessment  display 
should  not  be  affected  by  the 
'indicator  history  (grayed  out).  It 
needs  to  be  clear  to  the  user  that 
this  is  a  review  of  indicator 
assessments. 


As  justifications  move 
backwards  in  time,  the  indicator 
being  displayed  should  be 
highlighted,  and  its  data  reflect 
the  changes.  Scrolling  back  and 
forth  in  the  history  window 
should  allow  toggling  between 
the  older  and  newer  states/data. 

Subordinate  operation  roll¬ 
ups  should  not  be  affected 
hy  the  history  scroll-back. 
History  reviews  only 
indicator  justifications  and 
values.  In  fact,  these  roll¬ 
ups  should  be  grayed  out  to 
avoid  confusion. 


Justification  history  should  also  allow  attached 
files  used  for  justifications  to  be  opened  in 
another  window  for  review 


The  location  of  the  history  cursor  on  the  trend  line  should 
vary  with  scroll  control  -  start  out  on  current,  then  slide 
back  to  show  the  approximate  point  on  the  trend  line 
when  an  update  occurred. 


Figures?:  Indicator  History  Concept 


A  possible  method  of  display  for  multiple  collateral  indicators  is  depicted  in  Figure  38.  In  the 
concept  shown,  collateral  effect  bounds  and  assessed  points  are  shown  as  transparent  over-lays, 
with  the  degree  of  transparency  shown  by  the  order  in  the  stack.  The  currently  selected  collateral 
effect  area  is  displayed  as  a  solid  fdl.  Non-selected  areas  are  displayed  as  partially  transparent 
fills  behind  this  solid  layer.  Non-selected  assessed  points  are  depicted  as  partially  transparent 
foreground  symbols. 

With  these  characteristics,  this  has  a  weakness  in  obscuring  a  smaller  “background”  allowable 
effect  area  under  the  foreground  area,  but  preserves  the  “background”  assessment  “X”  to  indicate 
the  presence  of  another  assessment  for  review. 

General  Improvements  to  the  Operator  Interface.  Several  minor  changes  to  the  user  interface 
were  suggested  based  on  a  human  engineering  review.  These  include: 

1)  It  is  not  obvious  that  the  indicator  bars  on  the  right  side  of  the  display  can  be  double¬ 
clicked  to  adjust  an  indicator.  To  improve  their  affordance,  it  is  recommended  to  make 
them  single-click  to  open  and  give  them  some  mild  hover  effect  when  moused-over, 
possibly  a  background  color  change.  Incorporating  this  difference  will  also  let  the  user 
know  when  it  can  be  clicked  on  or  not  since  it  is  not  always  interact-able,  e.g.,  in  the  edit 
weight  set  window. 
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2)  Subordinate  operation  bars  on  the  right  side  of  this  display  behave  differently  than  the 
indicator  bars.  It  is  recommended  that  subordinate  operation  bars  should  be  represented 
somewhat  differently  to  avoid  confusion  by  the  operator.  A  separate  control  could  be 
provided;  however,  the  tradeoff  between  adding  an  additional  control  versus  increasing 
display  clutter  would  need  to  be  evaluated.  Additional  feedback  from  actual  operators  can 
refine  this. 

3)  The  operation  level  displayed  in  the  center  graphic,  whether  IFO  or  active  10,  should  be 
highlighted  in  the  Assessment  Explorer.  Currently,  the  prototype  initializes  by  showing  the 
summary  operation  level,  "Operation  Healing  Voice,"  but  this  is  not  highlighted  in 
Assessment  Explorer. 

4)  Remove  the  “X”  for  closing  the  window  from  the  "Edit  Weight  Set"  window  to  prevent  the 
operator  from  clicking  the  X  rather  than  using  the  Commit  Changes  button  and  thinking 
they  actually  made  changes. 


TT3D2:  Open  telecom  firewall 


The  currently  selected 
collateral  effect  (who’s 
indicators  are 
displayed  and  may  be 
altered),  is  displayed  as 
an  outlined  solid 
foreground  fill. 


“Background”  collateral 
effects  (effects  for  which 
indicators  are  not 
currently  displayed  on 
the  balance  of  the 
window),  are  depicted 
by  transparent 
underlays. 


Duration 


“Background” 
assessments  are 
displayed  using 
partially  transparent 
“X”  symbols  in  the 
foreground  of  the 
graphic.  They  should 
not  be  obscured  by 
the  foreground  fill  of 
the  selected  collateral 
effect. 


Clicking  a 
“background” 
assessment  symbol 
should  select  the 
collateral  effect  and 
bring  it  into  the 
foreground  and 
display  its  indicators. 


Figure  38:  Multiple  Collateral  Effects  Concept  for  Active  10 
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5) 


When  there  is  only  one  indicator,  the  edit  weight  selection  is  irrelevant;  thus,  it  is 
recommended  to  disable  the  edit  weight  selection  button  under  this  circumstance,  to  avoid 
confusion  and  frustration  on  the  part  of  an  operator  who  may  not  understand  why  the 
button  exists  but  does  not  work. 

6)  Alter  the  depiction  of  active  10  desired  or  collateral  effects  to  ovals.  Currently,  rectangular 
regions  are  used  to  display  desired  and  collateral  effects  bounds  in  terms  of  duration  and 
degree  of  effect.  It  is  likely  that  this  may  overstate  the  limits  of  these  bounds  near  the 
comers  of  the  rectangle  when  both  factors  are  approaching  their  limits.  An  alternate  to  this 
display  is  an  oval  region  defined  at  a  minimum  by  semi-major/minor  bounds  that 
correspond  to  duration/degree  of  effect.  However,  as  effects  bounds  are  not  necessarily 
symmetrical,  the  mathematically  attractive  solution  of  displaying  a  simple  ellipse  is 
insufficient.  This  is  an  area  for  mathematical  exploration  before  committing  to  graphics 
prototypes,  as  a  rigorous  solution  that  lends  itself  to  roll-up  calculations  may  prove  quite 
complex.  The  class  of  shapes  known  as  Cassini  ellipses  and  also  Super  ellipses  may  be 
useful,  though  the  rigorous  solution  probably  falls  into  the  complex  area  of  general 
elliptical  curves. 

7)  Provide  a  detailed  “analyst’s  view”  toggle  for  the  active  10  display  to  include  point  &  click 
editing  of  effects  bounds.  The  current  active  10  display  uses  multiple  x-axis  time  scales  on 
a  single  display  to  provide  at-a-glance  comparison  of  the  rough  durations  of  effects 
between  10  tasks.  However,  this  same  display  poorly  supports  the  precise  interpretation  of 
durations  by  an  analyst  -  at  the  right  side  of  the  display;  the  difference  of  a  few  pixels  is  a 
large  change  in  duration. 

This  display  should  be  able  to  be  toggled  to  a  single,  uniform  scale,  selected  by  the  user.  If 
the  collateral  area  on  the  scale  selected  extends  beyond  the  display  limits,  either  a  pan 
control  to  move  the  windowed  area  of  the  display  on  the  scale  or  a  scale  change  should  be 
implemented.  Implementing  this  “analysts’  view”  would  potentially  allow  a  click-and-drag 
interface  for  setting  or  modifying  effect  boundaries. 

8)  Provide  a  stand-alone  capability;  including  local  Op/effect  input  screens  and  database.  As 
alluded  to  in  the  paragraph  above,  a  stand-alone  capability  to  enter  at  least  a  rudimentary 
10  plan,  designating  effects  and  indicators,  is  desired  to  support  smaller  operations,  locally 
controlled  10  operations,  and  compartmented  activities.  Ideally,  this  interface  should  be 
simple,  with  immediate  feedback  of  the  result.  One  suggested  interface  is  to  start  with  a 
blank  tree  view  that  contains  a  single,  blank  “Operation.”  Right-clicking  on  this  blank 
operation  would  allow  a  user  to  enter: 

•  Descriptive  information  for  the  selected  element. 

•  Creation  of  a  new  operational  effect  subordinate  to  the  selected  level 
(tactical  objective,  IFO  or  10  tactical  task,  etc.). 

•  Creation  of  a  new  indicator  subordinate  to  the  selected  level. 

In  this  manner,  a  user  familiar  with  the  conventions  of  the  display  could  rapidly  construct  a 
small  plan  from  a  blank  tree,  populating  it  with  related  indicators  and  subordinate  tasks,  by 
building  and  describing  effects  and  indicators  subordinate  to  a  selected  level. 
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Paired  with  this  capability  would  be  a  locally  resident  database  capable  of  storing  both  the 
plan  and  its  history  data. 

9)  Allow  expected  magnitude  to  be  set  by  indicator,  rather  than  by  effect.  In  the  Despotistan 
scenario,  a  desired  IFO  effect  was  to  influence  the  government  to  allow  NGO  workers 
supported  by  foreign  military  to  enter  the  country  unmolested  by  either  national  troops  or 
local  militias.  Since  the  desired  effect  impacted  the  entire  country,  the  magnitude  assigned 
was  equal  to  the  population  (e.g.,  24,000,000).  However,  the  magnitude  indicators  used  to 
determine  the  effect  are  numerically  much  smaller:  detection  of  messages  urging 
cooperation  on  official  Despotistan  broadcast  channels  (e.g.,  five  channels  possible),  and 
observation  of  the  withdrawal  of  heavy  brigades  from  the  border  (e.g.,  twelve  brigades). 

In  this  case,  the  expected  magnitude  for  each  indicator  differs  from  the  expected  magnitude 
of  the  effect:  it  can  be  argued  that  this  is  not  an  exceptional  instance,  but  a  normal 
occurrence  in  IFO  assessment.  It  would  be  simpler  for  the  assessor  if  indicator-specific 
expected  magnitudes  could  be  assigned  and  displayed  for  each  indicator,  so  that  the 
assessor  is  entering  real- word  numbers  for  the  indicator  rather  than  scaling  the  value  in 
their  head:  the  software  can  scale  and  weight  the  indicators  for  display  in  the  overall  effect 
summary. 

Essentially,  IFO  assessment  can  be  characterized  as  describing  a  population  through  a  sort 
of  clustered  sampling  method,  using  one  or  more  sample  frames  and  sampling  weights. 
Thus  in  the  example  above,  we  are  attempting  to  describe  the  receptivity  of  the  population 
of  Despotistan  to  foreign  intervention  by  using  two  different  sampling  frames:  1)  the 
content  of  government-controlled  broadcasts,  and  2)  the  behavior  of  forward  based  troops. 
Within  these  frames,  the  assessor  will  need  to  articulate  a  sampling  strategy  (e.g..  When  do 
I  listen?  How  often  do  I  look?).  The  recommended  change  recognizes  that  these  two 
frames,  while  assumed  to  have  a  correlation  to  the  affected  population,  in  reality  have  their 
own  individual  magnitude.  With  this  approach,  if  the  assessor  can  fully  characterize  the 
size  and  distribution  of  the  sample  frame,  it  is  possible  to  evaluate  his  selected  sample 
strategy  and  assign  a  confidence  limit. 
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13.  Acronyms  and  Abbreviations 


ACC 

Air  Combat  Command 

ACUMEN 

Advanced  Capability  for 
Understanding  and  Managing 
Effects  Networks 

AECV 

Action-Effects  Contrast 
Visualization 

AFC2ISRC 

Air  Force  Command  and 
Control,  Intelligence, 
Surveillance,  and 
Reconnaissance  Center 

AFDC 

Air  Force  Doctrine  Center 

AFTTP 

Air  Force  Tactics, 
Techniques,  and  Procedures 

AFRL 

Air  Force  Research 
Laboratory 

AIR 

Assessment  Information 
Request 

AOC 

Air  Operations  Center 

ATD 

Advanced  Technology 
Demonstration 

ATF 

Air  Force  Assessment  Task 
Force 

ATO 

Air  Tasking  Order 

BDA 

Bomb  Damage  Assessment 

C2ISR 

Command  and  Control 
Intelligence  Surveillance  and 
Reconnaissance 

CART 

Combat  Automation 
Requirements  Testbed 

CFC 

Combined  Forces 

Commander 

CMMf^ 

Capability  Maturity  Model® 
Integration 

COA 

Course  Of  Action  Analysis 

CSA 

Cognitive  Systems  Analysis 

CSE 

Cognitive  Systems 
Engineering 

CWE 

Cognitive  Workflow  Element 

CWR 

Cognitive  Work 

Requirements 

DASEA 

Dynamic  Air  and  Space 
Effects-based  Assessment 

DMPI 

Desired  Mean  Point  of 
Impact 

DSE 

DMPI  -  Sortie  -  Equivalents 

EBA 

Effects-Based  Assessment 

EBAO 

Effects-Based  Approach  to 
Operations 

EBO 

Effects-Based  Operations 

EBOD 

Effects-Based  Operational 
Design 

EBP 

Effects-Based  Plan 

ESC 

Electronic  Systems 

Command 

FDA 

Functional  Damage 
Assessment 

FFDB 

Functional  Flow  Block 
Diagrams 

GCIC 

Global  Cyberspace 
Integration  Center 

GEM-S 

Global  Effects  Matrix- 
Synchronization 

GEV 

Geo-spatial  Effects 
Visualization 

GPN 

Goal  Process  Nodes 

lAM 

Indicator  Analysis  Manager 

IFO 

Influence  Operation 

10 

Information  Operations 

IRR 

Information  Relationship 
Requirements 

ISRD 

Intelligence  Surveillance  and 
Reconnaissance  Division 

IWPC 

Information  Warfare 
Planning  Capability 

JEFX 

Joint  Expeditionary  Forces 
Experiment 
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JFACC 

Joint  Forces  Air  Component 
Commander 

JFC 

Joint  Forces  Commander 

JFCOM 

Joint  Forces  Command 

JIOWC 

Joint  Information  Operations 
Warfare  Center 

JMEM 

Joint  Munitions  Effectiveness 
Manual 

MEA 

Mission  Effects  Assessment 

MoE 

Measures  of  Effectiveness 

MoP 

Measures  of  Performance 

MVC 

Model- View-Controller 

OA 

Operational  Assessment 

OEAVT 

Operational  Effect 
Assessment  Visualization 
Tool 

OAT 

Operational  Assessment 

Team 

OE 

Operational  Environment 

OLE 

Operational  Line  of  Effect 

00 

Operational  Objective 

PDA 

Physical  Damage  Assessment 

PDS 

Persistent  Data  Store 

PEL 

Primary  Effects  List 

PMESII 

Political,  Military,  Economic, 
Social,  Infrastructure,  and 
Information 

SCVOA 

System  for  Cognitive 
Visualization  of  Operational 

Assessment 

SDT 

Strategy  Development  Team 

SME 

Subject  Matter  Expert 

SoS 

Systems  of  Systems 

SPT 

Strategy  Plans  Team 

TA 

Tactical  Assessment 

TAC 

Tactical  Assessment  Cell 

TAT 

Targets  and  Assessment 
Team 

TBMCS 

Theater  Battle  Management 
Core  System 

TBONE 

Theater  Battle  Operations 
Net-centric  Environment 

TI3 

TBONE-IWPC  Indicator 
Interface 

TIM 

Technology  Impact  Matrix 

TLE 

Tactical  Line  of  Effect 

TO 

Tactical  Objective 

TRL 

Test  Readiness  Level 

TT 

Tactical  Tasks 

TTLE 

Tactical  Task  Line  of  Effect 

TTO 

Tactical  Task  Objectives 

TTP 

10  Tactics,  Techniques  and 
Procedures 

WAIT-C 

Warfighter  Analysis  of 
Innovative  Technologies  and 
Concepts  workshop 

XGen 

Next-Generation  Assessment 
Environment 
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APPENDIX  A 


OEAVT  CONCEPT  MAPS  AND  CRITICAL  DECISION  SPREADSHEETS 


A-l 


R^prnnri  mends 

a  ^ 


Target  System 
Assessment 


Conducted  by 


Future  Operation^  f 


Combatant 

Command 


Assesses 


Overall  Impact  on 
Systems  Capabilities 


Fuses 


With  Support 
From 


BOA  on  Functionat  Damage 


Others  Conducting 
TSA 


National  Level 
Assets 


!8 


coreistS'Of 


A-3 


A-4 


A-5 


recommend 
FD  parameters 


A-6 


A-7 


OperatiOfial 

Objectives 


Develop 

Objectives 


-input-to-^ 


Plans 

Team 


produces 


Develop 

Tactical 

Tasks 


— contributes'to-^ 


develop 

MOE_ 


— contributes'to 


I  develop 


contributes-to- 


Develop 
Causal  Linkages 


-resufts-in- 


A-8 


A-9 


Functional  Damage 


Assessment 


With  suppoft 


3  -.Jrom 
^  F.  r 


f - - ^ 

National  Level 

Assets 

\ _ _ _ J 


A-IO 


[tofllPetfcfwanct] 


PATt-of 


A-11 


PhhIuce^ 


SWKiStSHlrf 


A-12 


A-13 


A-14 


Provide  Continuous  OA  Inciuding  Predictive  Evaluations 


Assess 

operational 

level 

execution 


*SORTI  Status 


*WOE 


*Actual  vs  planned 


*Resources 

available/not 

available 


*Objectives 


*  Desired  effects 


*Mission 

successes/failures 


*Plan  vs  actual 


*Determine  if 
execution  is 
going  as 
planned 


*lnterpret  CA  sources  and 
summaries 


*Assess  mission  to  check  if  it 
requires  a  different  level  of  effort 


*The  system  shall  aid  in  determining  if  actual  actions 
are  taking  place  as  planned 


A-15 


Provide  Continuous  OA  Inciuding  Predictive  Evaluations 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Communicate 

with 


Assess 

operational 

level 

environment 


*Weather 


*Air  space 


*Change  in 
weather 


*Change  in 
terrain 


*Change  in  air 
space 


*Determine  if 
weather, 
terrain,  and/or 
air  spaces 
current/future 
will  affect  plan 


*lnterpret  information  from  psyops 


*lnterpret  information  from  ops 
analyst 


*lnterpret  information  from  combat 
ops  rep 


*lnterpret  information  from 
ISRD/ACF 


*lnterpret  information  from 
component  rep 


*lnterpret  information  from  combat 
plans  rep 


*  Interpret  information  from  space 
analyst 


*lnterpret  information  from  doctrine 
strat  expert 


*lnterpret  information  from  IW 
analyst 


*PSYOPS 

*Ops  Analysts 

*Combat  Ops  rep 

*ISRD/ACF 

*Component  Reps 

*Combat  Plans  Rep 

*Space  Analyst 

*Doctrine  Strat  Expert 

*IW  Analyst 

Collections  Manager 


*The  system  shall  aid  in  determining  if  weather,  terrain, 
and/or  air  spaces  current/future  will  affect  plan 
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Provide  Continuous  OA  Inciuding  Predictive  Evaluations 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Communicate 

with 


*lnterpret  information  from 
collections  manager 


Assess 

progress 

toward 

tactical 

objectives 


*Accomplishment 
status  of  each  TO 


*Accomplishment 
status  of  each  TT 


*SORTI  Status 


*Actual  vs  planned 


*Resources 

available/not 

available 


*Objectives 


*  Desired  effects 


*Mission 
failures  / 
successes 


*Determine  if 
tactical 
objectives  will 
be  achieved 


Assess  status 


*The  system  shall  aid  in  determining  if  tactical 
objectives  will  be  achieved 
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Provide  Continuous  OA  Inciuding  Predictive  Evaluations 


*Mission 

successes/failures 


A-18 


Provide  Continuous  OA  Inciuding  Predictive  Evaluations 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Communicate 

with 


Assess 

progress 

toward 

operational 

objectives 


*Accomplishment 
status  of  each  00 


*Command 
objectives  and 
intent 


*Desired  and 
expected  effects 


*Enemy  actions, 
plans  and 
capabilities 


*Aggregated  DM  PI 
and  SORTI  status 


*Mission 
failures  / 
successes 


*Determine  if 
operational 
objectives  will 
be  achieved 


*Assess  accomplishment  of  each 
00 


*Merge  information  to  address 
questions  relevant  to  achievement 
of  JFACC  objectives 


*Provide  evidence  to  other  strat 
team  members 


*Assess  extent  to  which  ongoing 
ops  are  achieving  command 
objectives  and  intent 


*Evaluate  ops  effectiveness  on 
enemy  action  and  capability 


*The  system  shall  aid  in  determining  if  operational 
objectives  will  be  achieved 
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AFOTTP 

1 

Update/tailor 
performance 
measures  to 
assess  JFACC 
objectives  and 
tasks  unique  to 
the  relevant  ATO 

*Objectives 

*Decide  when  measures 
need  to  be  updated  or 
changed 

*Determine  how  to 
change  them 

*Determine  if  the 
measures  or  the 
objectives  need  to  be 
changed 

*Update  / 
tailor 

measure 

*JFACC 

guidance 

*ATO 

*MOE,  MOP, 
Sis 

*The  system  shall  have 
access  to  Objectives 

*The  system  shall  have 
access  to  Tasks 

*The  system  shall  have 
access  to  ATO 

*The  system  shall  allow 
have  access  to  all 
measures  associated 
with  tasks  and  objectives 

*The  system  shall  allow 
changes  to  be  made  to 
measures 

AFOTTP 

2 

Update/tailor 
effectiveness 
measures  to 
assess  JFACC 
objectives  and 
tasks  unique  to 
the  relevant  ATO 

*Objectives 

*Decide  when  measures 
need  to  be  updated  or 
changed 

*Determine  how  to 
change  them 

*Determine  if  the 
measures  or  the 
objectives  need  to  be 
changed 

*The  system  shall 
provide  the  capability  to 
view  JFACC  objectives, 
tasks,  and  their 
associated  measures, 
and  the  AOD  all  at  once 
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Coordinate/Receive/Integrate  Inputs  from  CA  Sources  that  Support  OA 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Common  Toois 

Errors  used 


Commu 

nicate 

with 


Requirements 


Acquire  CA 
information 
when 
needed 


*Know  where 
information  is 
located 


*Know  source 
of  information 


*Know  time  of 
information 


*Know  what 
mission  CA  is 
for 


*Source 

*Timing 

*Accuracy 

*Reliability 


*Determine 
usefulness  of 
report 


*Determine  if  report 
lines  up  with  other 
report 


*Read  reports 


*Compare  to  other 
reports 


*Not  receive 
report 


*Not  receive 
report  in  timely 
manner 


*Web 


*Network 


Components 


Imagery 


The 

biggest 

problems 

are 

receiving 
the  data 
and 

receiving 
it  in  a 
timely 
manner 


*The  system  shall  provide  a  way  to 
receive  combat  assessment 
information 


*The  system  shall  provide  a  way  to 
request  information 


*The  system  shall  provide  a  way  to 
"alert"  people  that  a  new  report  is 
available 


Correlate 
CA  inputs  by 
mission 


*ATO  mission 
identifiers 


*Targets 

attacked 

(DMPIs) 


*Assets 
assigned  to 
dynamic  targets 


*Reason  why 
mission  failed 


*Determine  impact 
of  mission 
success/failure  for 
the  upcoming 
mission 


*MISREP 

Summaries 


*The  system  shall  aid  in  correlating 
CA  inputs  to  the  mission 
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*Mission 

failures 


*Reasons  why 
missions  failed 


Interpret  CA 
from  ISR 


*Emerging 

Opportunities 


*OGA  analysis 


*Sortie/DMPI 

Status 


*Mission 

successes 


*Mission 

failures 


‘Reasons  why 
mission  failed 


*DMPI  Success 
/  Failures 


‘Explanations 
of  missions 


‘Determine  if  a  re¬ 
roll  is  needed 


‘Determine  if 
mission  was  a 
success/failure 


‘Determine  if  a 
modification  to  the 
plan  is  needed 


‘Make 

recommendations 


‘ISR  reports 


‘The  system  shall  provide  access  to 
ISR  data  (summaries) 


‘The  system  shall  provide  a  way  to 
compare  data  and  rate  it  accordingly 
to  usefulness 
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Correlate 
CA  results 
with  TT,  TO 
and  00 


*Objectives 


*Priorities 


*Mission 

success/failures 


*CA  doesn't 
correlate  with 
other  data 


*CA  indicates 
effects  are  not 
being  achieved 


*Determine  if  WOE 
should  be 
maintained  or 
changed 


*Determine  if  data 
correlates  or 
contradicts  one 
another 


*Determine  if  we 
are  maintaining 
priorities 


*Determine  if  a 
modification  to  the 
plan  is  needed 


*Make 

recommendations 


*ISR  reports 


*Other 

summaries 


*The  system  shall  aid  in  correlating 
CA  results  with  TT,  TO,  and  00 


*The  system  shall  aid  in  determining 
if  objectives  are  being  met 


*The  system  shall  aid  in  determining 
if  WOE  should  be  maintained  or 
changed 


*The  system  shall  aid  in  determining 
if  data  correlates  or  contradicts  one 
another 


*The  system  shall  aid  in  determining 
if  we  are  maintaining  priorities 


*The  system  shall  aid  in  determining 
if  a  modification  to  the  plan  is 
needed 
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Assess 
Degree  of 
progress 
toward 
objectives 


*Plan 

*Determine  if 
objectives  are 
being  met 

*Determine  if  WOE 

*WOE 

*Time 

should  be 
maintained  or 
changed 

*Time  (day  and 

*WOE 

*Determine  if  a 
modification  to  the 

hours) 

plan  is  needed 

*Where  you  are 

*Actual  vs 
Planned 

*Determine  if  you 

actually 

are  ahead/behind 

compared  to 

plan 

plan 

*Determine  if 
another  day  needs 
to  be  added  to 
achieve  objectives 

*The  system  shall  display  progress 
towards  objectives 


*The  system  shall  display  the  Plan 


*The  system  shall  display  the  WOE 


*The  system  shall  display  the  Time 
(day  and  hours) 


*The  system  shall  display  where  you 
are  actually  compared  to  plan 


*The  system  shall  aid  in  determining 
if  objectives  are  being  met 


*The  system  shall  aid  in  determining 
if  WOE  should  be  maintained  or 
changed 


*The  system  shall  aid  in  determining 
if  a  modification  to  the  plan  is 
needed 


*The  system  shall  aid  in  determining 
if  you  are  ahead/behind  plan 


*The  system  shall  aid  in  determining 
if  another  day  needs  to  be  added  to 
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Coordinate/Receive/Integrate  Inputs  from  CA  Sources  that  Support  OA 


R 

e 

f 

e 

r 

e 

n 

c 

e 

C 

o 

d 

e 


achieve  objectives 
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Derive 

intended 

and 

unintended 
effects  from 
CA  results 


*Know  what  the 
intended  effects 
are 


*Adversary 

behaviors 


*System 

behaviors 


*Determine  if 
effects  are  being 


*Determine  what  is 
causing  the  effect 


*Determine  how 
the  effect  is  being 
achieved  and  how 
long  it  will  last 


*Request  more 
Intel  data 


*The  system  shall  provide  a  way  to 
derive  unintended  and  intended 
effects  from  CA  results 


*The  system  shall  aid  in  determining 
if  effects  are  being  achieved 


*The  system  shall  aid  in  determining 
what  is  causing  the  effect 


*The  system  shall  aid  in  determining 
how  the  effect  is  being  achieved  and 
how  long  it  will  last 


*MEA 


Monitor  CA 
information 
channels 


*Know  what  to 
monitor 


*Alert  that 
something  new 
is  available 


*Determine  what 
objective  the  report 
summarizes 


*Read  summary 


*Not  knowing 
that  something 
new  is  available 


*The  system  shall  provide  a 
capability  to  monitor  CA  channels 


*The  system  shall  organize  data 


*SITREPS 


A-26 


COA  Analysis/Wargaming  Consulting 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Common 

Errors 


Requirements 


*completeness 


*Feasibility 


*Form  their 

recommendation  to  the 
commander  on  which 
COA  option  to  adopt 


*Do  the  assumptions  have 
any  impact  on  attaining 
the  desired  effects? 


*May  modify  COA 


*Consider  all  facts  and 
assumptions  of  the 
estimate  and  their 
possible  effect  on  the 
action 


*The  system  shall  provide  the  capability  to  wargame 
COAs 


*The  system  shall  provide  a  basic  framework  for  the 
development  of  the  COAs 


*The  system  shall  "play"  the  outcome  of  COAs 


MM1, 

Reg 


Blue  COA  war 
gamed  against 
Red  COA 


*Probability  of 
friendly  losses 


*Time  to  attain 
objectives  and 
desired  effects 


'Collateral 

damage 


*lf  I  achieve  the  desired 
effect,  how  will  the  enemy 
respond? 


*How  likely  is  this 
reaction? 


*How  will  I  know? 


*Will  it  add  to  or  subtract 
from  my  objective? 


*Consider  conflict 
termination  issues. 
Think  through  own 
action,  adversary 
reaction,  and 
counteraction. 


*Assess  the  likelihood 
of  achieving  objectives 
and  desired  effects 
given  likely  enemy 
reactions  (Look  at  the 
World  through  the 
eyes  of  the  adversary) 


*The  system  shall  identify  weaknesses  in  the  COA 


*The  system  shall  display  friendly  losses 


*The  system  shall  display  both  desired  effects  and 
undesired  effects 


*The  system  shall  display  collateral  damages  based 
on  COAs 


*The  system  shall  display  options  to  let  user  choose 
the  best  COAs 
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COA  AnalysisAA^argaming  Consulting 


L 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Common 

Errors 


Requirements 


*lf  unintended  effects 
occur,  how  might  the 
enemy  respond? 


*How  likely  is  this 
reaction,  given  the 
unintended  effects? 


*How  will  I  know? 


*The  system  shall  provide  a  reporting  mechanism 


*The  system  shall  allow  modifications  to  COAs 


*How  will  allies/coalition 
partners  respond? 


*Why  do  I  believe  all  of 
the  above? 


MM1, 

REg 


When  COA  are 
modified, 
recalculates  the 
probability  of 
attaining 
commanders 
intent  as  well  as 
the  changes  in 
the  criteria 
values 


*know  how  to 
modify  COA 


*Significance  of 
changes 


*Determine  if  calculation 
indicates  COA  needs  to 
be  changed 


*How  changes 
affect  the  COA 
in  regards  to 
intent 


*Recommend  changes 
*Determine  how  and  enhance  the  COA 

where  COA  needs  to  be 
modified 


*The  system  shall  recalculate  the  probability  of 
attaining  commanders  intent  was  well  as  the 
changes  in  the  criteria  values  when  COAs  are 
modified 


*Determine  if  COA  needs 
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COA  AnalysisAA^argaming  Consulting 


L 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Common 

Errors 


Requirements 


to  be  thrown  out 


Reg 


Create  sequel 
plans  that  allow 
friendly  forces 
to  capitalize  on 
achievement  of 
objectives  and 
desired  effects 


*The  system  shall  allow  the  creation  of  sequel  plans 
that  allow  friendly  forces  to  capitalize  on 
achievement  of  objectives  and  desired  effects 


Reg 


Assess  the 
likelihood  of 
Undesired 
Effects 

occurring  given 
the  likely  enemy 
reactions 


*History  of 
adversary 


*Adversary 

allies 


*Adversary 

capabilities 


*Adversaries 

intent 


*Adversary 

behaviors 


*How  might  the  enemy  act 
to  produce  Undesired 
Effects? 


*Have  I  considered  the 
consequences  of  all 
known  Undesired  Effects? 


*Recommend  changes 
that  mitigate  the  risk  of 
causing  undesired 
effects 


*Create  basic  branch 
plans  that  address 
enemy  reactions  and 
mitigate  risks  of 
unintended  and 
undesired  effects 


*Not  get  data 
on  adversary 


*Not  "catch"  an 
unintended 
effect 


*The  system  shall  support  and  aid  in  the 
assessment  of  the  likelihood  of  Undesired  Effects 
occurring  given  the  likely  enemy  reactions 


*The  system  shall  provide  the  history  of  adversary 


*The  system  shall  provide  information  concerning 
the  adversary  allies 


*The  system  shall  provide  information  concerning 
adversary  capabilities 


*The  system  shall  provide  information  concerning 
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COA  AnalysisAA^argaming  Consulting 


Ref 

ere 

nee 

Co 

de 


CWR 


IRR 


Critical 
Cue  and/or 
Factors 


Critical  Decisions 


Actions 


Common 

Errors 


U 

s 

e 

d 


C 

o 

m 

m 

u 

n 


c 

a 

t 

e 

w 

it 

h 


D 

a 

t 

a 


P 

r 

o 

d 

u 

c 

t 

s 


Requirements 


adversaries  intent 


*The  system  shall  aid  in  determining  how  might  the 
enemy  act  to  produce  Undesired  Effects. 


*the  system  shall  aid  in  considering  the 
consequences  of  all  known  Undesired  Effects. 
*The  system  shall  recommend  changes  that  mitigate 
the  risk  of  causing  undesired  effects 


*The  system  shall  create  basic  branch  plans  that 
address  enemy  reactions  and  mitigate  risks  of 
unintended  and  undesired  effects 
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COA  AnalysisAA^argaming  Consulting 


Ref 

ere 

nee 

Co 

de 


CWR 


IRR 


Critical 
Cue  and/or 
Factors 


Critical  Decisions 


Actions 


Common 

Errors 


U 

s 

e 

d 


C 

o 

m 

m 

u 

n 

c 

a 

t 

e 

w 

it 

h 


D 

a 

t 

a 

P 

r 

o 

d 

u 

c 

t 

s 

u 

s 

e 

d 


Requirements 


*  Desired  effects 


*Undesired 

effects 


*Determine  if  COA  path  is 
correct 


*The  system  shall  aid  in  the  visualization  on  how 
operations  will  unfold  based  on  the  selected  COA 


MM1 


Visualize  how 
operations  will 
5  unfold  based  on 
the  selected 
COA 


*lntentions 


*History  of 
adversary 


*What  part  of 
COA  is  not 
working 


*Determine  if  COA  path 
needs  to  modified 


*Play  COA  out  from 
start  to  end 


'Determine  if  COA  will 
accomplish  mission 


*Evaluate  steps  within 
the  COA 


*Adversary 

allies 


*Adversary 

capabilities 


*Determine 

undesired/desired  effects 
along  pathway 


*The  system  shall  aid  in  determining  if  COA  path  is 
correct 


*The  system  shall  aid  in  determining  if  COA  path 
needs  to  modified 


*The  system  shall  aid  in  determining  if  COA  will 
accomplish  mission 


*The  system  shall  aid  in  determining 
undesired/desired  effects  along  pathway 


*Adversaries 

intent 
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COA  AnalysisAA^argaming  Consulting 


L 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Common 

Errors 


Requirements 


AFOTTP 


Ensure  COA 
options  attain 
commanders 
intent 


*Commanders 

Intent 


*Timing 


*Deadlines 


*Resources 


*Determine  if  COAs  have 
measurements  that  can 
be  collected 


*Determine  COAs  meet 
standards  (such  as 
timing) 


*Develop  COAs 
that  do  not  have 
measurements 
that  can  be 
gathered  for 
assessment 


*The  system  shall  aid  in  the  comparison  of  COA 
options  to  commanders  intent  to  ensure  intent  is 
being  met 


*The  system  shall  aid  in  determining  if  COAs  are 
measurable  for  assessment 


*  Desired  effects 


Reg 


*Timing 


Use  previously 
developed 
items  for 
analysis 


*Prediction  of 
enemy 
reactions 


‘Unintended 

effects 


'History  of 
artifacts 


‘Successful  / 
failures 


'Determine  the  need  to 
search  for  archives 


‘Not  know  that 
archives  exist 


‘Determine  which 
archives  are  useful 


‘Search  for  archives 


‘Evaluate  archives 


‘Not  know 
where  to  find 
archives 


‘Determine  how  to  search 
for  archives 


‘Not  able  to  find 
archives 


‘Rationale  for 
decisions  made 


‘The  system  shall  provide  access  to  archival  data 
concerning  past  mission 


‘The  system  shall  provide  a  way  to  search  for 
archives 


‘The  system  shall  provide  a  way  to  archive  data 


‘The  system  shall  provide  tips  on  how  to  search  for 
archives 
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COA  AnalysisAA^argaming  Consulting 


L 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Common 

Errors 


Requirements 


Reg 


Build  a  timeline 
to  identify  when 
certain 

objectives  are 
projected  to 
occur 


*Determine  when  events 
should  occur 


*Determine  when  events 
should  be  complete 


*Determine  what  events 
precede  others 


*The  system  shall  aid  in  building  a  timeline  to 
identify  when  certain  objectives  are  projected  to 
occur 


*The  system  shall  plot  and  aid  in  determining  when 
events  should  occur 


*The  system  shall  plot  and  aid  in  determining  when 
events  should  be  complete 


*The  system  shall  plot  and  aid  in  determining  what 
events  precede  others 


Reg 


Identify 

advantages  and 
disadvantages 
of  each  COA 
based  on 
wargaming 


*The  system  shall  aid  in  identifying  advantages  and 
disadvantages  of  each  COA  based  on  wargaming 
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COA  AnalysisAA^argaming  Consulting 


L 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Common 

Errors 


Requirements 


Reg 


10 


Refine  each 
COA  based  on 
COA 

wargaming 


*Modify  each  COA 
based  on  enemy’s 
most  likely  and  most 
dangerous  reactions 
based  on 

recommendations, 
sequel  and  branch 
plans 


*Validate  FrOB  based 
on  COA  wargaming--if 
better  results  could  be 
achieved  with  a 
different  force  mix, 
recommend  such  a 
FrOB. 


*The  system  shall  allow  refinement  of  each  COA 


*The  system  shall  allow  to  modification  of  COAs 

*The  system  shall  aid  in  developing  enemy’s  most 
likely  and  most  dangerous  reactions  based  on 
recommendations,  sequel  and  branch  plans 


*The  system  shall  aid  in  validating  FrOB  based  on 
COA  wargaming 


*The  system  shall  recommend  a  different  force  mix. 


*Rank  resources 
according  to  expected 
contributions  of 
friendly  and  adversary 
forces  to  achieving 
desired  effects  with 
less  risk  of  failure  or 
greater  probability  of 
success 


*The  system  shall  aid  in  and  allow  the  ranking  of 
resources  according  to  expected  contributions  of 
friendly  and  adversary  forces  to  achieving  desired 
effects 


AFOTTP  1 1 


Build  a  timeline 
to  identify  when 
certain  tasks 
(actions)  are 
projected  to 


*The  system  shall  aid  in  build  a  timeline  to  identify 
when  certain  tasks  (actions)  are  projected  to  occur 
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COA  AnalysisAA^argaming  Consulting 


L 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Common 

Errors 


Requirements 


Build  a  timeline 
to  identify  when 
certain  effects) 
are  projected  to 
occur 


*the  system  shall  aid  in  building  a  timeline  to  identify 
when  certain  effects)  are  projected  to  occur 


*  Desired  effects 


*Timing 


AFOTTP 


13 


Develop  a 
linkage  to 
support  a  timely 
and  accurate 
analysis 


*Prediction  of 
enemy 
reactions 


*Unintended 

effects 

*Rationale  for 
decisions  made 


*The  system  shall  aid  in  developing  linkage(s)  to 
support  a  timely  and  accurate  analysis 


*Timeline  when 
objectives, 
tasks,  and 
effects  will  take 
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COA  AnalysisAA^argaming  Consulting 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Common 

Errors 


Requirements 


place 
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Ref 

ere 

nee 

Co 

de 

# 

CWR 

IRR 

Critical  Cue 
and/or 
Factors 

Critical 

Decisions 

Actions 

Co 

m 

mo 

n 

Err 

ors 

Tools 

used 

Co 

m 

mu 

nic 

ate 

wit 

h 

Dat 

a 

Pro 

du 

cts 

use 

d 

Com 

ment 

s 

Requirements 

AFOTTP 

1 

Receive  JFACC 
guidance  and 
objectives 

*The  system  shall  allow  access 
to  guidance  and  objectives 

AFOTTP 

2 

Develop 

guidance 

*Objectives 

*  Desired  effects 

*Adversary 

behavior 

*Friendly  resources 

*The  system  shall  provide  the 
ability  to  develop  guidance  and 
objectives 

AFOTTP 

3 

Develop  TO 

*Objectives 

*  Desired  effects 

*Determine  and 
develop 
objectives  that 
will  satisfy 
overall 
objectives 

*Attend 

meeting 

*Brainstorm 

*Whiteboard 

*Paper 

*Other 

AOC 

divisions 

*The  system  shall  allow 
generation  of  Tos 

*The  system  shall  provide  a 
template  to  develop  Tos 

*Determine  how 
many  objectives 
are  needed 

*Develop 

TO 

*Word  SW 

*The  system  shall  provide  a  way 
to  save  the  objectives 

*Objectives 

*Determine  if 

measures  are 

collectable 

*The  system  shall  allow 
generation  of  MOEs 

AFOTTP 

4 

Develop  MOEs 
for  each  TO 

*  Desired  effects 

*T0 

*Determine  if 

measures  are 
"good" 
measures 

*Evaluate 

measures 

*The  system  shall  provide  a  way 
to  save  MOEs 

*The  system  shall  provide  a  way 
to  access  old  MOEs 

AFOTTP 

5 

Develop  00 

*Objectives 

*  Desired  effects 

*Determine  and 
develop 
objectives  that 
will  satisfy 
overall 
objectives 

*Attend 

meeting 

*Brainstorm 

*Whiteboard 

*Paper 

*Other 

AOC 

divisions 

*The  system  shall  allow 
generation  of  Oos 

*The  system  shall  provide  a 
template  to  develop  Oos 

*Guidance 

*Determine  how 
many  objectives 
are  needed 

*Develop 

00 

*Word  SW 

*The  system  shall  provide  a  way 
to  save  the  objectives 
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COA  Comparison/Selection 


Critical  Cue 
and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


*Objectives 

*The  system  shall  allow 
generation  of  Sis 

AFOTTP 

6 

Develop  Si's  for 
each  00 

*  Desired  effects 

*Determine  if 
the  Sis  are 
"good" 
indicators 

*Evaluate 

Sis 

*The  system  shall  provide  a  way 
to  save  Sis 

*00 

*The  system  shall  provide  a  way 
to  access  old  Sis 

*Objectives 

*  Desired  effects 

*Determine  if 
tasks  will 
achieve 
objective 

*The  system  shall  allow 
generation  of  TTs 

AFOTTP 

7 

Develop  TT 

*Resources 

available 

*Adversary 

system 

information 

*Adversary 

behaviors 

*Adversary 

resources 

*Adversary  intent 

*Determine  risk 
with  tasks 

*Determine  how 
effective  tasks 
will  be 

*Develop 

tasks 

*Brainstorm 

*Attend 

meeting 

*Whiteboard 

*Paper 

*Word  SW 

*Other 

AOC 

divisions 

*IPB 

*INSUM/ 

DISUM 

*TPFDD 

*The  system  shall  allow  access 
to  TPFDD 

*The  system  shall  allow  access 
to  Adversary  system  information 
(IPB,  DISUM,  INSUM) 

*Friendly 

system 

information 

*Determine 
amount  of  time 
tasks  will  take 

*The  system  shall  provide  a 
summary  of  friendly  forces 
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COA  Comparison/Selection 


ere 

Critical  Cue 

Critical 

Decisions 

nee 

Co 

# 

CWR 

IRR 

and/or 

Factors 

de 

*Objectives 

‘Determine  if 

*  Desired  effects 

‘Friendly  intent 

measures  can 

be  collected 

*COA 

‘Determine  if 

AFOTTP 

8 

Develop  MOPs 
for  each  TT 

*Resources 

‘Friendly  resources 

measures  are 
"good" 

Available 

‘Time 

‘Determine  how 

*lntent 

long  it  will  take 
to  collect 

measures 

‘Tactical  tasks 

‘Objectives 

‘Adversary 

behaviors 

‘Determine 
which  COA  will 
best  achieve 

COA 

comparison  and 
selection 

objectives 

AFOTTP 

9 

‘Desired  effects 

‘Adversary 

resources 

‘Determine 

‘Adversary 

systems 

‘Adversary  intent 

which  COA 
produces  less 
risk 

Actions 


*Develop 

measures 


*Evaluate 

measures 


Requirements 


*The  system  shall  allow 
generation  of  MOPs 


*The  system  shall  aid  in 
determining  if  measures  can  be 
collected 


*The  system  shall  aid  in 
determining  if  measures  are 
"good" 


*The  system  shall  aid  in 
determining  how  long  it  will  take 
to  collect  measures 


*Compare 

COA 


*Rank  COA 


*The  system  shall  allow 
comparison  of  multiple  COAs 


*The  system  shall  aid  in 
determining  which  COA  will  best 
achieve  objectives 


*The  system  shall  aid  in 
determining  which  COA 
produces  less  risk 


*the  system  shall  allow  the 
comparison  of  COA 


*The  system  shall  allow  the 
ranking  of  COA 
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COA  Comparison/Selection 


Critical  Cue 
and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


Evaluate  COA 
analysis  / 
wargaming 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  provide 
various  methods  to  analyze 
multiple  COAs 


Reg 


Operational 
objectives  are 
prioritized 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow 
operational  objectives  to  be 
prioritized 


Reg 


12 


Operational 
objectives  are 
sequenced 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow 
operational  objectives  to  be 
sequenced 


Reg 


13 


Operational 
objectives  are 
phased 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow 
Operational  objectives  to  be 
phased 
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COA  Comparison/Selection 


Critical  Cue 
and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


Reg 


Operational 
objectives 
weight  of  effort 
determined 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*Resources 

available 


*The  system  shall  allow 
Operational  objectives  weight  of 
effort  to  be  determined 


Reg 


TO  are 
prioritized 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow  TO  to  be 
prioritized 


Reg 


16 


TO  are 
sequenced 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow  TO  to  be 
sequenced 


Reg 


TO  are  phased 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow  TO  to  be 
phased 
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COA  Comparison/Selection 


Critical  Cue 
and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


Reg 


TO  weight  of 
effort 

determined 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*Resources 

available 


*The  system  shall  allow  TO 
weight  of  effort  to  be  determined 


Reg 


TT  are 
prioritized 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow  TT  to  be 
prioritized 


Reg 


20 


TT  are 
sequenced 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow  TT  to  be 
sequenced 


Reg 


TT  are  phased 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*The  system  shall  allow  TT  to  be 
phased 


A-42 


COA  Comparison/Selection 


Com 

merit  Requirements 

s 


Reg 

22 

TT  weight  of 
effort 

determined 

*Objectives 

*  Desired  effects 

*Adversary 

systems 

*Resources 

available 

*TPFDD 

*The  system  shall  allow  TT 
weight  of  effort  to  be  determined 
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COA  Comparison/Selection 


Critical  Cue 
and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


Reg 


23 


Determine 
desired  effects 


*Priority  of 
effects 


*Sequencing  of 
effects 


*Weight  of  effort 


*Duration  of 
effect 


*The  location  of 
where  effects 
needs  to  take 
place 


*Timing  aspect  of 
attaining  the  effects 


*Priority  of  effects 


*Sequencing  of 
effects 


*Weight  of  effort 


*Duration  required 
of  effect 


*Place  of  effect 


*Why  do  I 
believe  the 
actions  taken 
will  result  in  the 
desired  effects? 


‘Likelihood 
effects  will 
attain 
Objectives 
(Why  do  I 
believe  this?) 


*How  will  I 
know  when 
effects  are 
achieved 
(MOE) 


*What  reaction 
do  I  expect  from 
the  enemy  and 
why 


*What 

Indicators  will 
identify  success 
or  failure  in 
attaining  the 
effect(s) 


*Why  do  I 
believe  all  of 
the  above 
(rationale) 


‘Timing 
aspect  of 
attaining 
desired 
effects  is 
provided  by 
focusing  the 
prioritization, 
sequencing, 
phasing, 
and  weight 
of  effort, 
such  as  to 
attain  them 
at  the  time, 
place  and 
duration 
required 


‘The  system  shall  display  the 
priority  of  effects 


‘The  system  shall  display  the 
sequencing  of  effects 


‘They  system  shall  display  the 
Weight  of  effort 


‘The  system  shall  display  the 
duration  of  effect 


‘The  system  shall  display  the 
location  of  where  effects  needs 
to  take  place 
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use 
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Com 

ment 
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Requirements 


Reg 

24 

Determine 

potential 

unintended 

effects 

‘Priority  of 
effects 

‘Sequencing  of 
effects 

‘Weight  of  effort 

‘Duration  of 
effect 

‘The  location  of 
where  effects 
needs  to  take 
place 

‘Timing  aspect  of 
attaining  the  effects 

‘Likelihood  of 
them  occurring 
and  why 

‘Impact  on 
JFACC/JFC 
Objectives 
(positive  and 
negative) 

‘If  negative 
impacts  how 
can  they  be 
avoided 

‘How  will  1 
know  if/when 
unintended 
effects  occur 
with  what 
Indicators 

‘What  risk  is 
associated  with 
the  unintended 
effects?  Is  it 
acceptable? 

‘Not 
receive 
data  in 
timely 
manner 

‘Intel 

data 

‘Other 

mission 

reports 

‘The  system  shall  aid  in 
determining  potential  unintended 
effects 

Reg 

25 

Determine 

comparison 

criteria 

‘Objectives 

‘Desired  effects 

‘Adversary 

systems 

‘Resources 

available 

Criteria 
usually  will 
be 

determined 
by  the 
JFACC,  but 
suggested 
criteria 
include  risk 
(of  success 
and  of 
failure), 
timeliness, 
and 

operability 

‘The  system  shall  aid  in 
determining  comparison  criteria 
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Reg 

26 

Rate  each  COA 
against  criteria 

*Criteria 

*Determine  if 
COA  meets 
criteria 

*The  system  shall  allow  the 
ability  to  rate  each  COA  against 
the  criteria 

Reg 

27 

Recommend 
highest  rated 
COA 

*Criteria 

*Objectives 

*  Desired  effects 

*ls  there  any 
reason  why  1 
should  choose 
a  lower-rated 
COA? 

If  so,  there  may 
be  other 
comparison 
criteria  that 
should  be 
considered. 

*The  system  shall  allow  the 
ability  to  recommend  the  highest 
rated  COA 

Reg 

28 

Refine  COA 
based  on 
JFACC/JFC 
decision  and 
guidance 

*Criteria 

*Objectives 

*  Desired  effects 

*COA 

*Adversary 

systems 

*Determine  how 
to  modify  COA 

*Modify 

COA 

*The  system  shall  provide  the 
ability  to  modify  COA 

29 

Aid  in  the 
development  of 
the  STTM 

*COA  adequacy 

*Forces 

required 

*Risk 

*The  system  shall  provide  the 
ability  to  develop  the  STTM 
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*Time 

*Completeness 

*Feasibility 

*Probability 
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COA  Development 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Requirements 


MM1 


Plan  or  sequence 
of  action  is  how 
the  goal  or 
objective  is  going 
to  be  accomplish 


*Objectives 

*  Desired  effects 

*Adversary 

behavior 

*Adversary 

systems 

*Adversary 

intent 

*Friendly 

systems 

*Adversary 

resources 

*Resources 

available 

*COG  Critical 
capabilities 

*COG  Critical 
requirements 

*COG  Critical 
Vulnerabilities 

*Fielded  forces 

*lnfrastructures 

*Determine  if  sequence  of 
actions  will  effectively 
achieve  objective 


*Determine  if  an  "event' 
needs  to  be  modified  to 
better  achieve  objectives 


*Modify  plan 


*The  system  shall  provide  a  plan  or  sequence  of  actions 
needed  to  achieve  the  goal 


*The  system  shall  provide  a  way  to  modify  plan 


Reg 


Analyze  and 
identify  friendly 
COGS 


*Determine  if  COG  is 
friendly  or  hostile 


*The  system  shall  provide  the  ability  to  analyze  and  identify 
friendly  COGs 


*The  system  shall  provide  the  ability  to  identify  COG  Critical 
capabilities 


*The  system  shall  provide  the  ability  to  identify  COG  Critical 
requirements 


*The  system  shall  provide  the  ability  to  identify  COG  Critical 
Vulnerabilities 
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COA  Development 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Requirements 


*Political 


Reg 


Develop  Effects 
(desired  and 
undesired) 


*Objectives 


*  Desired  effects 


*Adversary 

systems 


*Friendly 

systems 


*Resources 

available 


*Why  do  I  believe  these 
effects  will  attain  the 
operational  objectives 


*How  will  these  effects 
support  attaining  the 
objectives? 


*How  will  I  know  these 
effects  have  been 
achieved? 


*Why  do  I  believe  these 
effects  will  occur? 


*Will  these  other  effects 
add  to  or  subtract  from 


*Assess  the 
likelihood  of  the 
desired  effects 
attaining  the 
objective.  This 
requires  an 
understanding  of 
the  cause  and 
effect  relationship. 


*Recommend 
success  indicators 
(MOEs)  that  will 
aid  in  assessment 
of  desired  effects 
(ISR  Strategy  and 
planning — third 
pillar  of  PBA) 


*The  system  shall  provide  the  capability  to  assess  the  likelihood 
of  the  desired  effects  attaining  the  objective. 


*The  system  shall  provide  the  capability  to  recommend  success 
indicators  (MOEs)  that  will  aid  in  assessment  of  desired  effects 
(ISR  Strategy  and  planning — third  pillar  of  PBA) 


*The  system  shall  provide  the  capability  to  Determine  potential 
unintended  effects 


*The  system  shall  provide  the  capability  to  Assess  the  likelihood 
of  all  unintended  effects  occurring?  Assess  the  impact,  or  value, 
of  the  unintended  effects  with  respect  to  the  JFACC’s  and  the 
JFC’s  objectives 


*The  system  shall  provide  the  capability  to  Recommend 
indicators  that  will  aid  in  assessment  of  unintended  effects  (ISR 


A-49 


COA  Development 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Requirements 


accomplishing  the 
objectives? 


*How  can  1  avoid 
undesired  effects? 


Can  I  modify  the  action  or 
take  some  other  action  in 
addition  to  the  original 
action? 


*Howwill  I  know  when 
unintended  effects  have 
occurred? 


*Why  do  I  believe  these 
effects  will  attain  the 
tactical  objectives? 


*How  will  the  actions 
support  achieving  these 
effects? 


*Determine 

potential 

unintended  effects 


*Assess  the 
likelihood  of  all 
unintended  effects 
occurring? 


Assess  the 
impact,  or  value, 
of  the  unintended 
effects  with 
respect  to  the 
JFACC’s  and  the 
JFC’s  objectives 


*ln  cases  where 
the  impact  is 
counterproductive, 
recommend 
actions  that  might 
mitigate  the  risk  of 
undesired  effects 


*Recommend 
indicators  that  will 


Strategy  and  planning — third  pillar  of  PBA) 


*The  system  shall  provide  the  capability  to  Determine  the 
tactical  objectives  that  will  accomplish  operational  objectives. 
Determine  desired  effects  (direct  and  indirect)  that  will  achieve 
objectives 


*The  system  shall  provide  the  capability  to  Assess  the  likelihood 
of  the  desired  effects  attaining  the  objective.  This  requires  an 
understanding  of  the  cause  and  effect  relationship  (causal 
linkage) 


*The  system  shall  provide  the  capability  to  Determine 
supporting  actions/tasks 


*The  system  shall  provide  the  capability  to  Refine  COAs  based 
on  priority,  sequence,  phasing,  weight  of  effort,  and  matched 
resources 


*The  system  shall  provide  the  capability  to  Ensure  Phases  (and 
objectives)  support  JFC  phasing,  objectives,  and  end  state 


*The  system  shall  provide  the  capability  to  Sequence  Tasks 
(Actions)  to  accomplish  objectives  and  to 
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COA  Development 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Requirements 


aid  in  assessment 
of  unintended 
effects  (ISR 
Strategy  and 
planning — third 
pillar  of  PBA) 


*Determine  the 
tactical  objectives 
that  will 
accomplish 
operational 
objectives. 
Determine  desired 
effects  (direct  and 
indirect)  that  will 
achieve  objectives 


*Assess  the 
likelihood  of  the 
desired  effects 
attaining  the 
objective.  This 
requires  an 
understanding  of 


*The  system  shall  provide  the  capability  to  Identify  risk  areas  for 
each  COA 
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COA  Development 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Requirements 


the  cause  and 
effect  relationship 
(causal  linkage) 


*Determine 

supporting 

actions/tasks 


*Refine  COAs 
based  on  priority, 
sequence, 
phasing,  weight  of 
effort,  and 
matched 
resources 


*Ensure  Phases 
(and  objectives) 
support  JFC 
phasing, 
objectives,  and 
end  state 


*Sequence  Tasks 
(Actions)  to 
accomplish 
objectives  and  to 


*ldentify  risk  areas 
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COA  Development 


Critical 

Cue  and/or  Critical  Decisions 
Factors 


Actions 


Requirements 


for  each  COA 
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Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


AFOTTP 


Develop  air 
and  space 
assessment 
measures  for 
assessing 
JFACC 
objectives  and 
tasks  during 
execution 


*JFACC  objectives 


*JFACC  tasks 


*Resources  that 
will  perform 
missions 


*Adversary  history 


*Desired  effects 


*Undesired  effects 


*Determine 

assessment 

measures 


*Develop 

assessment 

measures 


*The  system  shall  support  in  developing  assessment  measures  for 
missions 


What,  where, 
how  joint  force 
will  affect  the 
enemy  or  the 
situation  at 
hand 


*COA 


*Resources  being 
used 


*Adversary 

capabilities 


*Adversary 

intentions 


*Area 

surroundings 


*Buildings 
located  in 
space 


*Other 

environmental 
features  in  area 


*Determine  the 
effectiveness  of  the 
COA  in  terms  of 
specified  measures 
that  meet  the 
objectives  and 
guidance 


*The  system  shall  aid  in  the  prediction  of  events  that  may  occur  during  a 
COA  (play  out) 


*The  system  shall  provide  a  graphical  representation  of  the  area 
*The  system  shall  have  access  to  adversary  data 


Determine  end 
state  — what  is 
to  be 

accomplished. 


*Desired  effects 


*Undesired  effects 


*Determine  which 
COA  will  best 
accomplish  the  end 
state 


*The  system  shall  provide  a  means  to  input  end  states  and  aid  in  the 
development  of  COAS  to  achieve  end  state 
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Coordinate  with  ISRD 


*The  system  shall  provide  access  to  or  a  way  to  input  the  purpose  or 
rational  as  to  why  the  goal  is  sought 


Determine  the 
plan  or 
sequence  of 
actions  of  how 
the  goal  or 
objective  is 
going  to  be 
accomplished 


*Adversary 

systems 


*Friendly  systems 


*Weather 


*Geography 


*Determine  the 
sequence  of 
actions  to  achieve 
goal 


*Develop 

plan(s) 


*The  system  shall  provide  visually,  a  plan  or  sequence  of  actions  on  how 
the  goal  or  objective  is  going  to  be  accomplished 


*The  system  shall  provide  access  to  environmental  information  such  £ 
weather. 


Specify  the 
resources  for 
the  plan 


*Resources 

available 


*SPINS  or  other 
rules 


*Determine  the 
"best"  resource  for 
a  particular  task 


*Allocate 
resources  to 
tasks 


*Not 

have 

updated 

TPFID 


*The  system  shall  display  a  lisitng  of  available  resources  for  the  plan  (e.g. 
TPFID) 


Determine  info 
required  to 
support  OA 


*Types  of 
information 
available 


*"Who"  and  "how" 
will  the  information 
be  gathered 


*Time 

*Accuracy 

*Reliability 


*Determine  the 
types  of 

information  needed 
for  assessment, 
and  when  the 
information  is 
needed 


*Determine  who  to 
ask  for  the 
information  needed 


*Request  for 
information 


*The  system  shall  aid  in  determining  what  info  is  required  to  support  OA 

*The  system  shall  allow  the  user  to  request  information 
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Coordinate  with  Special  Technical  Ops 


r 


Critical 

Cue  Critical 

and/or  Decisions 

Factors 


Actions 


Common  Tools 

Errors  used 


Requirements 


*Mission 

*COMMS 

Identify  who 
you  need  to 
communicate 
with 

*Need  to  know 
who  (positions) 
are  needed  to 
be  part  of  your 
team 

*Location  of 
team 
members 

*Criticality  of 
communicati 

*Decide  on 
priorities  of 
communication 
s  with  various 
team  members 

*Contact  team 
members  to  join 
communications 

*Not  contact  or 
know  if  a  critical 
team  member 

(phone, 

red 

phone) 

*Email 

*Need  to  know 
how  to  get  in 
contact  with 
team  members 

ons 

*Chat 

*Need  to  know 

*Choose  wrong 

what  comms 
are  working/not 
working 

*Decide  on 
working  status 
of  comms 

comm  mode  for 
mission 

Choose  proper 
communicatio 
n  mode 

*Need  to  know 
the  "best" 
comm  mode  for 
current  job 

*Various 

comm 

methods 

*Decide  what 
the  best  comm 
mode  is  for  the 
job 

*Choose  a 
comm  that  is 
not  working 
properly 

*Comm 

modes 

*Choose  a 

*Need  to  know 

comm  mode 

*Org 

chart 


*The  system  shall  provide  access  to  mission 


*The  system  shall  provide  a  list  of  team  members 


*The  system  shall  provide  a  means  to  contact  team  members 


*The  system  shall  provide  various  communication  pathways 

*The  system  shall  provide  ways  to  test  comms 

*The  system  shall  provide  a  diagnosis  of  comms 


*The  system  shall  aid  in  the  decision  making  of  choosing  a 
comm  mode  for  mission 
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Coordinate  with  Speciai  Technical  Ops 


that  does  not 
work  for  every 
team  members 


*The  system  shall  display  various  communication  pathways 


Ensure 
communicatio 
n  pathway  is 
working 


*Need  to  know 
how  to  test 
comms 


*Need  to  know 
procedures  to 
test  COMMS 
status 


*Need  to  know 
how  to  fix 
comms  if  not 
working 
properly 


*Noise  /  no 
noise 


*No 

response 


*Decide  on  best 
method  to  test 
comms 


*Decide  on  best 
method  to  fix 
comms 


*Fix  comms 


*Test  comms 


*Not  test 
properly 


*Not  know  how 
to  fix 


*Misdiagnose 
the  problem 


*The  system  shall  provide  methods  to  test  comms 


*The  system  shall  provide  a  diagnosis  of  problems  with 
comms 


*The  system  shall  provide  checklists  on  how  to  fix  problem 
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Coordinate  with  Speciai  Technical  Ops 


Ensure  team 
members  are 
using  same 
pathway 


*Need  to  know 
if  team 
members  are 
"hooked"  into 
the  correct 
comm  pathway 


*Need  to  know 
who  is  part  of 
your  team 


*Need  to  know 
who  you  need 
to  communicate 
with 


*No 

response 


*The  system  shall  provide  status  of  who  is  on-line 


*The  system  shall  provide  a  team  summary 


Ensure  a 
second  comm 
method  is 
chosen 


*Need  to  know 
what  the  next 
best  method  for 
communication 
is 


*Need  to 
ensure  comm  is 
working 


*Type  of 
mission 


*Team 
responsiven 
ess  to 
particular 
comms 


*Decide  on  best 
comm  method 


*Decide  if 
comm  is 
working 
properly 


*Choosing  bad 
comm  method 


*The  system  shall  provide  listings  of  various  comm  modes 
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Coordinate  with  Speciai  Technical  Ops 


Set  up 

communicatio 
n  pathway 


*Need  to  know 
how  to  alert 
team  members 
what  the  comm 
pathway  is 


*Need  to  know 
how  to  set  up 
comms  so  the 
constant 
comms  can  be 
running 


*Set  comm 
pathway  up 
wrong 


*Not  alert  team 
members  what 
pathway  is 


*The  system  shall  enable  users  to  set  up  comm  pathways 


*The  system  shall  allow  a  alerts  to  let  team  members  know 
what  the  comms  pathway  being  used  is 


Coordinate 
with  team 
concerning 
where  and 
how  often  data 
/  information 
will  be  stored 
or  sent 


*Know  available 
places  to  store 
information 


Communicate 
the  wrong 
information, 


*The  system  shall  allow  storage  of  information  in  a  database 
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Develop  Diagnostic  Assessment 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


Generate 
options  and 
recommendation 
for  strat  division 
consideration 


*  Desired 
effects 


*Unmeasurable 

COA 


*Determine  when  a 
modification  to  the 
plan  needs  to  be 
made 


*Suggest  changes 
with  alternatives 


Strat 

Team 


*The  system  shall  provide  a  means  to  develop  options  to  strat 
plans  and  compare  courses 


Compare  actual 
to  expected 
results  for 
tactical  tasks 


*  Desired 
effects 


*Undesired 

effects 


‘Unintended 

effects 


‘Mission 

failures/succes 


‘Adversary 

behavior 


‘Determine  if  plan 
was  successful 


‘Compare  plan  vs. 
actual  events 


Intel  data 


‘The  system  shall  allow  the  comparison  of  actual  to  expected 
results  for  tactical  tasks 
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Critical 
Cue  and/or 
Factors 


Develop  Diagnostic  Assessment 


Critical 

Decisions 


Actions 


Requirements 


Identify  reasons 
for  shortfalls 


*Mission 


*  Desired 
effects 


*Undesired 

effects 


*Unintended 

effects 


*Adversary 

behavior 


*Mission 

successes/fail 

ures 


*WOE 


*Adversary 

behavior 


*Mission 

failures 

reasoning 


*Determine  what 
the  shortfalls  are 


*Determine  a  way 
to  overcome 
shortfalls 


*Determine  if  a 
modification  to  the 
plan  is  needed 


*ldentify  shortfalls 


*Make 

recommendation 


*The  system  shall  identify  shortfalls  in  missions  and  plans 


*The  system  shall  support  the  decision  making  of  overcoming  the 
shortfalls 


*The  system  shall  aid  in  determining  if  a  modification  to  the  plan  is 
needed 
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Develop  Diagnostic  Assessment 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


Assess 
feasibility  of 
corrective 
actions 


*Resources 

available 


*Actual  vs 
Plan 


*WOE  for 
each  objective 


*Time  needed 
for  each 
objective 


*Time 


*Determine  if 
resources  available 
for  corrective 
actions 


*Determine  amount 
of  change  to  plan, 
and  what  to 
change 


*Determine  WOE 
for  each  objective 


*Determine  the 
time  needed  to 
achieve  each 
objective 


*Determine 
priorities  for  next 
day  according  to 
actual  vs  planned 


*Make 

recommendation 


*The  system  shall  aid  in  and  allow  the  assessment  of  the 
feasibility  of  the  corrective  actions 


*The  system  shall  determine  if  resources  are  available  for 
corrective  actions 


*The  system  shall  determine  amount  of  change  to  plan,  and  what 
to  change 


*The  system  shall  determine  the  WOE  for  each  objective 


*The  system  shall  determine  the  time  needed  to  achieve  each 
objective 


*The  system  shall  determine  the  priorities  for  next  day  according 
to  actual  vs  planned 
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Develop  Diagnostic  Assessment 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Requirements 


Predict  effects  of 
corrective 
actions  on 
tactical  and 
operational 
objectives 


*Resources 

available 


*Actual  vs 
Plan 


*WOE  for 
each  objective 


*Time  needed 
for  each 
objective 


*Priorities 


*lntended 

effects 


*Adversary 

systems 


*Friendly 

systems 


*Determine  if 
corrective  actions 
will  produce 
undesired  effects 
and  desired  effects 


*"Play"  out  COA 


*The  system  shall  allow  and  aid  in  the  prediction  of  effects  of  the 
corrective  actions  on  tactical  and  operational  objectives 


*The  system  shall  allow  and  aid  in  the  prediction  of  the  effects  that 
the  corrective  actions  on  tactical  and  operational  objectives  will 
have 
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Critical 
Cue  and/or 
Factors 


Develop  Diagnostic  Assessment 


Critical 

Decisions 


Actions 


Requirements 


Rank  potential 
corrective 
actions 


*lntended 

effects 


*Objectives 


*Adversary 
and  friendly 
systems 


*Resources 

available 


*Environmental 

conditions 


*Adversaries 
current  status 


*Determine  which 
plan  will  "best" 
accomplish 
objectives/effects 


*Determine  which 
plan  has  less  risk 
associated  with  it 


*The  system  shall  determine  which  plan  will  "best"  accomplish 
objectives/effects 


*The  system  shall  determine  which  plan  has  less  risk  associated 
with  it 


Communicate 
corrective 
actions  to  plans 
team 


*Know  who  to 
send 

modifications 

to 


*Know  proper 
format  of 
recommendati 
ons 


*Know  what 
information  to 
put  in  report 


*Know  how  to 
send  report 
out 


*Determine  when 
to  give  report 


*Determine  what 
information  to  put 
in  the  report 


*Determine  how 
long  the  report 


*Communicate  the 
report 


Power  Point 


*The  system  shall  allow  and  aid  in  the  selection  of  one  or  more 
corrective  actions 


*The  system  shall  provide  a  template  for  reports 


*The  system  shall  allow  and  aid  in  the  communication  of  the 
corrective  actions  to  the  plans  team 
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Critical 
Cue  and/or 
Factors 


Develop  Diagnostic  Assessment 


Critical 

Decisions 


Actions 


Requirements 


10 


Identify 

unanticipated 

operational 

limitations 


*Resources 

available 


*Actual  vs 
Plan 


*WOE  for 
each  objective 


*Time  needed 
for  each 
objective 


*Priorities 


*lntended 

effects 


*Changes  in 
priorities 


*Changes  in 
objectives  WOE 


*Changes  in 
resource 
availability 


*Determine  what 
limitations  will  have 
on  achieving 
overall  effect 


*Determine  how  to 
overcome 
limitations 


*The  system  shall  aid  in  determining  what  limitations  will  have  on 
achieving  overall  effect 


*The  system  shall  aid  in  determining  how  to  overcome  limitations 


A-65 


Develop  Overall  OA  Plan 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Common 

Errors 


Requirements 


AFOTTP 


Develop  an 
overall 
Operational 
Assessment  plan 
to  evaluate  both 
JAOP  planning 
and  actual 
execution  of  joint 
air  operations 


*The  system  shall  provide  the  ability  to  develop  an  OA 
plan 


*The  system  shall  provide  the  ability  to  compare  and 
evaluate  the  OA  plan  against  the  JAOP 


*The  system  shall  provide  the  ability  to  evaluate  OA  in 
real  time,  during  execution 


*The  system  shall  provide  the  ability  to  modify  the  OA 
plan  in  real  time 


*The  system  shall  aid  in  the  decision  making  of  where 
and  how  the  plan  should  be  modified 


Build  contact  list 
showing  links  and 
impacts  of  specific 
key  organizations 
on  the  OA 
process 


*The  system  shall  provide  the  capability  to  develop 
team  member  list(s) 
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Develop  Overall  OA  Plan 


Critical 
Cue  and/or 
Factors 


Critical 

Decisions 


Actions 


Common 

Errors 


Requirements 


AFOTTP 


Develop  an 
Operational 
Assessment  plan 
for  the  JAOC 


Plan  includes: 


*Evaluation  of 
JAOP  planning 
and  execution 
that  includes 
multiple 
sources 


Collection 
strategies,  and 
procedures  for 
gathering  and 
exploiting 
enemy 

intelligence  and 
friendly 
information 
(ISR,  10,  space, 
operations, 
logistics, 
comms) 


*Determine  best 
collection 
strategies  for 
mission(s) 


*Develop  plan 


*Distribute  plan 


*The  system  shall  allow  for  the  creation  of  an 
assessment  plan 


Integrate  JFC  and 
JFACC  OAs 


*lntegrate  JFC  and 
JFACCs  OA's  into 
the  plan 


*The  system  shall  allow  the  integration  of  both  JFC  and 
JFACC  OAs 


Evaluate  air, 
space,  and  10 
effectiveness  and 
efficiency 


*Determine 
effectiveness 
and  efficiency 
of  plans 


*Determine  if 
changes  need 
to  be  made  to 
plans 


*Review  and 
evaluate  plans 


*Air 

Component 


*Space 

Component 


*The  system  shall  allow  for  and  support  the  evaluation 
of  air,  space,  and  10  effectiveness  and  efficiency 


Establish 
relationships, 
contacts  to 
integrate 
functional  JAOC 


*Know  who  is 
part  of  the 
process 


*Experience  of 
individuals 


*lndividuals 


*Determine  who 
is  an  integral 
team  member 


*Strat  Team 


*Make  contacts 


*The  system  shall  allow  and  support  the  capability  to 
share  information  among  team  members 


*The  system  shall  allow  and  support  the  capability  to 
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Develop  Overall  OA  Plan 

r 

Ref 

ere 

nee 

Co 

de 

# 

CWR 

IRR 

Critical 
Cue  and/or 
Factors 

Critical 

Decisions 

Actions 

Common 

Errors 

T 

o 

o 

1 

s 

u 

s 

e 

d 

Com 

muni 

cate 

with 

Dat 

a 

Pro 

du 

cts 

use 

d 

Requirements 

activities  into  OA 
process 

*Know  what 
roles  each 
person  plays 

knowledge  of 
mission 

*ISRD 

integrate  data  received  from  team  members  into  the 
overall  OA  plan 

*Know  chain  of 
command 


*Amount  of  time 
in  the  area 


AFOTTP  7 


Determine 

assessment 

information 

requirements 


*The  system  shall  support  and  allow  the  development  of 
assessment  information  requirements 


Review: 


*JAOP 


*JFC  Objectives 


*JAOP 


*MAAP 


*The  system  shall  support  evaluation  of  JFACC 
objectives 


AFOTTP 


Evaluate  the 
effectiveness  of 
8  JFACC  objectives 
in  supporting  JFC 
objectives 


'Resources 

available 


*Credibility  and 
reliability  of 
products  and 
their  information 


'Strength  of 
support 


*Determine 
each  products 
effectiveness  in 
supporting  the 
overall 
campaign 


*MAAP 

*ATO 


*Weaponeered 
Source  JIPTL 


*Not  receive 
data  that  is 
needed  to  do 
an  effective 
evaluation 


*ATO 


*The  system  shall  allow  access  to  JAOP,  MAA,  ATO, 
JIPTL,  and  OPLAN 


"Weapon 

eered 

Source 

JIPTL 


*The  system  shall  allow  the  ability  to  request  the:  JAOP, 
MAA,  ATO,  JIPTL,  and  OPLAN 


*The  system  shall  aid  in  the  decision  making  and 
evaluation  between  product  and  objectives 


*OPLAN 


AFOTTP 


Identify  and 
establish  definitive 
linkages  to  any 
information  or 
intel  sources  that 
9  can  support 
combat  and 
Operational 
Assessment 
functions  within 


*Types  of 
information 
available 


*What  is 
actually 
available  to  the 
team  directly  / 
indirectly 


*Determine 
which  sources 
are  important  / 
not  important 


*Create/set 

linkages 


*OPLAN 


up 


*Not  identify  or 
know  that  an 
important  link 
exists 


*The  system  shall  support  and  aid  in  the  identification 
and  establishment  of  any  definitive  linkages  to 
information  or  intel  sources  that  can  support  combat 
and  Operational  Assessment  functions  within  the  JAOC 


the  JAOC 
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Develop  Overall  OA  Plan 


T 

Ref 

ere 

nee 

Co 

de 

# 

CWR 

IRR 

Critical 
Cue  and/or 
Factors 

Critical 

Decisions 

Actions 

Common 

Errors 

o 

o 

1 

s 

u 

s 

e 

Com 

muni 

cate 

with 

Dat 

a 

Pro 

du 

cts 

use 

d 

1 

1 

1 

Requirements 

d 


AFOTTP 


Refine  standard 
operation 
procedures 
specific  to 
individual  duties 
and 

responsibilities 


*Know  duties 
and 

responsibilities 
associated  with 
a  role 


*Know  roles 
needed  for 
mission(s) 


*Not  know  if 
role  is  needed 


*lndividual 

experience 


*Determine  who 
will  be  assigned 
to  what  role 


"Assign  roles  and 
responsibilities 


*Misassign 

responsibility 


*Workload 


*Time  frame 


'Determine  how 
many  tasks  to 
give  to  one 
individual 


"Assign  SOP  to 
individuals 


*Not  assign  a 
responsibility 


*The  system  shall  allow  for  the  refinement  of  standard 
operation  procedures  specific  to  individual  duties  and 
responsibilities 


*The  system  shall  allow  assignment  of  SOP  to 
individuals 


*The  system  shall  allow  assignment  of  roles  and 
responsibilities  to  individuals 


'Know  the  SOP 
for  each  role 


*Not  know  of  all 
SOPs 


*The  system  shall  aid  in  the  identification  of  SOPs, 
roles,  and  responsibilities 


*Adversary 

information 


AFOTTP 


Determine 
information 
requirements 
associated  with 
assessments 


'Past  mission 
information 


*Current 

mission 

information 


*Type  of 
mission 


*Objectives 


"Time  frame  to 
achieve 
objectives 


*Determine 

critical 

information 

requirements 


*Prioritize 

information 

requirements 


*Not  identify  a 
information 
requirement 


*Not  receive 
information 


*The  system  shall  allow  input  for  and  aid  in  the 
determination  of  information  requirements  associated 
with  assessments 


*Overall 

objectives 


*Know  where 
data  may  exist 


AFOTTP 


Obtain  information 
or  intel  to  support 
OA 


*Know  proper 
division  to  ask 
for  the  various 
types  of  data 


*Determine 
"best"  type  of 
intel  data  to  be 
used 


*Not  have 
access  to  data 


*Receive  data 


*Data  does  not 
exist 


ISRD 


*The  system  shall  allow  the  team  to  obtain  and  request 
information  or  intel 
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Integrate  OA  from 
JFACC  with  its 


AFOTTP 


counterparts  at 
the  JFC  level  to 
1  ensure  a  cohesive 
3  picture  between 
the  campaign  plan 
and  the  air  and 
space  portion  of 
that  campaign 


*During  OIF,  a 
senior  Operational 
Assessment 
analyst  was 
imbedded  into  the 
JFC’s  Campaign 
Operations 
Assessment  Board 
in  the  J8  staff.  This 
facilitated 

communication  and 
synchronization  of 
assessment 
activities  up 
through  HHQ. 


*Air 

reports 


*The  system  shall  allow  the  integration  of  the  OA  from 
JFACC  with  its  counterparts  at  the  JFC  level 


*Space 

Reports 


The  system  shall  aid  in  the  assessment  of  the  OA  Plan 
to  ensure  a  cohesive  picture  between  the  campaign 
plan  and  the  air  and  space  portion  of  that  campaign 


*lntegrate  air  and 
space  component 
OA  with  JFC 
counterparts 


*Determine  loss 


*Determine  cost 


The  system  shall  aid  and  support  in  determining  loss 


AFOTTP 


Determine  risk 


*COAs 


*  Desired  effects 


*Determine  if 
COA  is  routine 

or  not  *Perform  analysis  / 

evaluation 


*Ensure  that 
standards  for 
routine  events 
are  adequate  to 
provide  an 
acceptable  level 
of  risk. 


The  system  shall  aid  and  support  in  determining  cost 


The  system  shall  aid  and  support  in  determining  if  COA 
is  routine  or  not 


The  system  shall  aid  and  support  in  ensuring  that 
standards  for  routine  events  are  adequate  to  provide  an 
acceptable  level  of  risk. 
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Requirements 

*Risks 

*Determine  if 
plans  meet 
objectives 

*Balancing  risk  vs 
benefits 

*The  system  shall  allow  and  aid  in  the  balancing  of  risk 
vs  benefits 

AFOTTP 


1  Determine 
5  benefits  of  plan 


*Objectives 


*Plan  does  not 
meet  objectives 


*Determine  if 
benefits 
outweigh  risks 


*Eliminate 
unnecessary  risks. 


*Underestimate 

risk 


*The  system  shall  aid  in  and  allow  to  eliminate 
unnecessary  risks. 


*Know  available 
controls  to 
reduce  risk 


*Determine 
what  factors 
can  be  used  to 
control  risk 


*Reduce  the 
magnitude  of 
mission  essential 
risks  by  applying 
controls 


*The  system  shall  aid  in  an  allow  to  reduce  the 
magnitude  of  mission  essential 
risks  by  applying  controls 


1  Review  and  refine 
6  plan 


*Determine 
decision  points 
where  the 
commander  is 
required  by  the 
higher  guidance 
to  formally 
assess 
progress  and 
consider 
adjusting  the 
plan 


*Determine  if 
lack  of  effects 
achievement  is 
due  to 

inappropriate 
actions  or 
monitoring 
wrong  MOE 


*Evaluate  roles  and 
responsibilities  of 
components, 
coalition  members, 
and  the  DIE 
agencies  in  the 
EBA  process 


*Evaluate 

intelligence 

collection 

requirements 


*Evaluate  battle 
rhythm  to  track  and 
continuously 
review  the  MOE 
and  MP 


*Evaluate 
methodology  to 
examine  areas  of 
progress,  lack  of 
progress,  and 
causality 


*The  system  shall  allow  for  the  evaluation  of  roles  and 
responsibilities  of  components,  coalition  members,  and 
the  DIE  agencies  in  the  EBA  process 


*The  system  shall  allow  for  the  evaluation  of  intelligence 
collection  requirements 


*The  system  shall  allow  for  the  evaluation  of  battle 
rhythm  to  track  and  continuously  review  the  MOE  and 
MP 


*The  system  shall  allow  for  the  evaluation  of 
methodology  to  examine  areas  of  progress,  lack  of 
progress,  and  causality 


*The  system  shall  for  the  evaluation  of  the  methodology 
to  determine  whether  lack  of  effects  achievement  is  due 
to  inappropriate  actions  or  monitoring  wrong  MOE 
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*Evaluate 
methodology  to 
determine  whether 
lack  of  effects 
achievement  is  due 
to  inappropriate 
actions  or 
monitoring  wrong 


MOE 


ISRD 


Re 

Co 

m 

me 

nts 

fer 

en 

ce 

Co 

# 

CWR 

IRR 

Critical 
Cue  and/or 
Factors 

Critical 

Decisions 

Actions 

Common 

Errors 

Tools 

used 

Commu 

nicate 

with 

Data 
Product 
s  used 

Requirements 

de 

*Adversary  Activity 

*Adversary 

strategy 

*Are  ACFs 

*The  system  shall  provide  access 

*Adversary  status 

predictions  of 
adversary  behavior 
valid? 

ISRD-personnel 

expertise 

to  IPB  assessment 

AFOTTP 

1 

ISRD  provide 
assessment  of 

IPB,  adversary 
COAs,  COGs, 
Targeting,  and  ISR 
operations 

in  battlespace 

*Analysis  on  how 
ISR  determined 
strategic/operation 
al  advantage  and 
instantiate  over  the 
adversary 

*Adversary 

intention 

*Adversary 
desired  end 
state 

*Are  ISRD's  (ACF's 
and  Targets/CA's) 
estimate  of 
adversary  COGs 
still  valid? 

Provide 

recommendations 
to  modify  strategy 
to  better 
accomplish 
JFC/JFACC 
objectives  and 
desired  effects 

PowerPoint 

— ACF 
Strategist 

— Targets 
strategist 

*ACF 

Operational 

assessment 

documents 

(briefings, 

reports) 

*The  system  shall  provide 
adversary  COAs 

*The  system  shall  provide  access 
to  COG  information 

*The  system  shall  provide 
targeting  information 

*Determine 

*Level  of  success 
achieved  by  the 
ACF  at  acquiring, 
defining,  and 

*Adversary 
perception  of 
friendly 

effectiveness  of  the 
ACF  sections  of 
the  JAOP,  AOD, 
RSTA  Annex  in 
terms  of  objective 

— ISR  ops 

Strat 

*The  system  shall  provide  access 
to  ISR  operations 
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Critical 

Decisions 


Actions 


de 


closing  gaps  in  the 


Vulnerabilities 


PB 


*Adversary 

intentions 


and  task 
accomplishment, 
adherence  or 
divergence  from 
the  established 
plan,  and  optimum 
use  of  available 
resources 


ISRD 


Commu  Data 

nicate  Product  Requirements 

■ii-  .me  ^ 

with  s  used  . 


Common  Tools 

Errors  used 
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Cue  and/or 
Factors 

Critical 

Decisions 

Actions 

Common 

Errors 

Tools 

used 

Commu 

nicate 

with 

Data 
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de 

AFOTTP 

2 

Targeting 
specialist  provides 
targeting  expertise 
to  the  OA  process 

*Timeliness  and 
formatted  CA 
— BDA,  MEA,  MA 
(portions) 

*Review  BDA/MEA 
summaries 

*Not  get 
information  in 
timely  manner 

Targeting 

Strategist 

ISR  uses 

*BDA 

*MEA 

*MA 

TS 

serves 

as  the 
link 

between 
the  CA 
cell  in 
the  ISRD 
and  the 
OAT 

*The  system  shall  provide 

access  to  CA 

*The  system  shall  provide  access 
to  BDA 

*The  system  shall  provide  access 
to  MEA 

*The  system  shall  provide  access 
to  MA 

*The  system  shall  provide 
comms  between  OAT  and 
Targeting  specialist 

*The  system  shall  provide 
updates  of  status  of  CA,  BDA, 
MEA,  and  MA 

AFOTTP 

3 

Evaluate  targeting 
strategy  for 
effectiveness  in 
meeting 
JFC/JFACC 
objectives  and 
desired  effects 

*JFC/JFACC 

Objectives 

*JFC/JFACC 
Desired  Effects 

*Targeting  strategy 

*  Desired  effects 

*Undesired 

effects 

*Determine 
effectiveness  of 
targeting  strategy 
compared  to 
JFC/JFACC 
guidance 

*Evaluate  targeting 
strategy 

*Miss  critical 
information 

*  Incorrectly 
interpret 
information 

Targeting 

Strategist 

*Targeting 
strategy  report 

*The  system  shall  provide  access 
to  JFC/JFACC  objectives,  and 
desired  effects 

*The  system  shall  provide  access 
to  targeting  strategy 

*The  system  shall  provide  views 
to  compare  objectives,  and 
desired  effects  to  targeting 
strategy 
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Product 
s  used 

Requirements 

de 

AFOTTP 

4 

Promulgate 
targeting  strategy 
for  effectiveness  in 
meeting 
JFC/JFACC 
objectives  and 
desired  effects 

*JFC/JFACC 

Objectives 

*JFC/JFACC 
Desired  Effects 

*Know  who  to 
promulgate 
strategy  to 

*Ways  to 
promulgate 
strategy 

*Decide  when 
strategy  is  ready 

*Decide  who  needs 
to  know 

*Promulgate  to 
proper  teams 

*Not  ensuring 
that  OAT 
requirements 
are  clearly 
stated  and 
understood 

Targeting 

Strategist 

*The  system  shall  provide  the 
capability  to  develop  a  targeting 
strategy 

*The  system  shall  provide  a 
means  to  promulgate  plan  to 
team 

AFOTTP 

5 

Review  BDA/MEA 
summaries  to 
ensure  BDA/MEA 
is  factored  into  MA, 
future  target 
development  / 
nomination  and  OA 

*MA 

*Target  information 

*Operational 

Assessment 

*Ensuring  that 
BDA/MEA  is 
factored  into  MA, 
target  dev/nom  and 
OA 

*Ensure  that 
BDA/MEA  is 
factored  in 

Targeting 

Strategist 

ISR  uses 

*BDA 

*MEA 

*The  system  shall  provide  the 
capability  to  compare  BDA/MEA 
to  MA  to  ensure  accuracy 

AFOTTP 

6 

Evaluate  BDA 
results  for 
unintended  effects 

*lntended  effects 

*Desired  effects 

*Adversary 

capabilities 

*Mission  failure 

*Adversary 

behavior 

*Adversary 

capabilities 

*Determine 
unintended  effects 

*Evaluate  BDA 

*Misinterpret 

information 

*Not  receive 
data  in  timely 
manner 

*Not  receive 
data 

Targeting 

Strategist 

BDA 

*The  system  shall  provide  the 
capability  to  evaluate  BDA  for 
effects 

*The  system  shall  provide  data  to 
mission  success,  failures, 
shortcomings 

*The  system  shall  provide  access 
to  adversary  history,  capabilities, 
resources,  and  intentions 

*The  system  shall  provide  ways 
to  update  team  on  status  of  BDA 
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AFOTTP 

7 

Make  re-attack 
recommendations 

*Operational  level 
objectives 

*Mission  success 

*Mission  failures 

*Mission 

shortcoming 

*Priorities 

*Objectives 

*Desired  effects 

*Actual  effects 

*Resources 

available 

*Timing 

*Undesired 

effects 

*Determine  if  re¬ 
attack  is  needed 

*Make 

recommendation 

Targeting 

Strategist 

*The  system  shall  provide  means 
to  make  a  re-attack 
recommendation 

AFOTTP 

8 

ISR  ops  strat  and 
OAT  work  with  ISR 
ops  teams  and 

PED  Management 
team  to  obtain 
JAOC  ISR 
operations 
assessment  and 
refining 
subsequent 
JFACC  ISR 
strategy 

*JAOC  operations 
assessment 

*JFACC  ISR 
strategy 

*Determine  what  to 
refine  and  how  to 
refine  it. 

*ISR  Ops 
Strategist 

*PED 

Management 

*ISR  Ops 
Teams 

AFOTTP 

9 

Assesses  the 
results  of  TPED 
(Tasking, 
Processing, 
Exploitation,  and 
Dissemination) 

*Need  to  have 
TPED  data 

*Results  vs 
planned 

*Mission 
accomplished 
vs  not 

accomplished 

*Adversaries 

behaviors 

*Determine  what 
TPED  indicates 

*Evaluate  TPED 

*lncorrectly 

evaluate 

ISR  Ops 
Strategist 

TPED 

*The  system  shall  provide  access 
to  TPED 
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AFOTTP 

1 

0 

Refine  the  ISR 
strategy 

*ISR  Strategy 

*Mission 

*00 

*T0 

*TT 

*Determine  what 
changes  to  make 
to  strategy 

*Make 

modifications  to 

ISR  strategy 

ISR  Ops 
Strategist 

*The  system  shall  provide  access 
to  ISR  strategy 

*The  system  shall  allow 
modifications  to  ISR  strategy 

AFOTTP 

1 

1 

Measure  mission 
operational  factors 

*Platform 

*Sensor 

*Crew 

*Link  issues 

*Target  deck 
satisfaction 

*Resources 

down 

*Resources 

unavailable 

*Effectiveness 

ISR  Ops 
Strategist 

AFOTTP 

1 

2 

MOEs  to 
determine  if 
PIRs/ISR  tasks  are 
being  answered 

*MOEs 

*PIR/ISR  Tasks 

*ISR  Ops 
Strategist 

*ACF 

‘Targets 

Strategist 

*The  system  shall  provide  access 
to  PIR/ISR  tasking 

AFOTTP 

1 

3 

Receive  ISR  input 
to  the  OAT  plan  to 
evaluate  JAOP 
planning  and 
execution 

*ISR  Reports 

*BDA 

*  Effects 

ISR  Ops 
Strategist 
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AFOTTP 

1 

4 

Evaluate  the  ISR 
strategy  for 
effectiveness  in 
meeting  overall 

ISR  requirements, 
JFC/JFACC  PIRs 
and  supporting 
JFC/JFACC 
strategy  and  plans 

*Determine  if  ISR 
strategy  meets  all 
of  the  requirements 

*Evaluate  ISR 
strategy 

ISR  Ops 
Strategist 

*The  system  shall  provide  a 
means  to  evaluate  ISR  strategy 
by  evaluating  it  against  PIRs, 
strategies,  and  plans 

AFOTTP 

1 

5 

Monitor  the  ISR 
strategy  for 
effectiveness  in 
meeting  overall 

ISR  requirements, 
JFC/JFACC  PIRs 
and  supporting 
JFC/JFACC 
strategy  and  plans 

*Monitor  ISR 
strategy 

ISR  Ops 
Strategist 

*The  system  shall  provide  the 
ability  to  monitor  ISR  objectives 

AFOTTP 

1 

6 

Receive  ISR  report 
on  sections  of  the 
JAOP  and  AOD 
and  RSTA  Annex 
effectiveness 

*Objective 

*Task 

accomplishment 

*Adherence 

*Divergence  from 
the  established 
plan 

*Optimum  use  of 
available  resources 

*Receive  report 

ISR  Ops 
Strategist 

*ISR  Report 

*The  system  shall  provide  access 
to  ISR  report 

AFOTTP 

1 

7 

Receive  ISR 
operations 
assessment 
documentation 

*Receive  report 

ISR  Ops 
Strategist 

*ISR 

documentation 

*The  system  shall  provide  access 
to  ISR  documentation 
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AFOTTP 

1 

8 

Receive  data  from 
operations  and 
operational  results 

*ISR  platform 
*Distributed  PED 
performance 

*Problems  and 
limitations 

Collection 
satisfaction  for  pre¬ 
planned  collection 
decks 

*Timeliness 

*Operational 
effectiveness/succ 
ess  of  time 
sensitive  collection 
re-taskings 

Successes  / 
failures 

*  Effects 
(desired, 
undesired, 
unanticipated) 

*Receive  data 

*Not  receive 
data 

ISR  Ops 
Strategist 

*SITREPS 

*MISREPS 

*Other  reports 

*The  system  shall  provide  access 
to  multiple  sources  of  reports 

AFOTTP 

1 

9 

Receive  data  if  PIR 
is  answered 

*lf  the  PIR  is 
partially  answered 

*lf  the  PIR  is  not 
answered 

*Rational  for  the 
assessment 

*What  is  the 
recommendation  to 
best  answer  the 

PIR 

*Are  their 
associated  ISR 
task  proving 
effective 

*Determine  what 
state  PIR  is  in,  is  it 
answered,  not 
answered . 

*Evaluate  PIR 

ISR  Ops 
Strategist 

*The  system  shall  allow 
evaluation  of  PIR 
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AFOTTP 

2 

0 

Evaluate  MOEs  for 
ISR  tasks 

*Status:  not 
started,  in 
progress,  or 
completed 

*Effectiveness  of 
the  tasks:  not 
effective,  partially 
effective,  or 
effective 

*Status  offer 
each  MOE 

*Determine  if  MOE 
has  been  met  and 
successful 

*Evaluate  MOE 

ISR  Ops 
Strategist 

*The  system  shall  allow 
evaluation  of  MOEs 

ISR  uses 

*Desired  effects 

*IMINT 

AFOTTP 

2 

1 

Assessment  of 

Intel  data 

*Undesired  effects 

*Determine 
accuracy  of  all  data 
compared  to  Intel 
data 

*Evaluate  Intel  data 

ISR  Ops 
Strategist 

*SIGINT 

*The  system  shall  allow 
evaluation  of  Intel  data  compared 
to  other  sources  of  data 

*Missions 

*MASINT 

*HUMINT 

A-80 


JAOP  Development  Consulting 


AFOTTP 


Recommend 
changes  to  JAOP 


*Objectives 


*  Desired  effects 


*Status  of 
adversary 


*Status  of 
friendly 


*Determine  how 
effectively/thoroughly 
JAOP  supports  campaign 
plan 


*Determine  how 
thoroughly  and  effectively 
it  supports  the  overall 
theater  campaign  plan 
from  the  perspective  of 
operational  assessment 


*Evaluate  Air,  space, 
and  10  effectiveness 
and  efficiency 


*JAOP 


*lntel 

data 

*DISUM/ 

INSUM 


*The  system  shall  aid  in  determining  how 
effectively/thoroughly  JAOP  supports 
campaign  plan 


*The  system  shall  aid  in  determining  how 
thoroughly  and  effectively  it  supports  the 
overall  theater  campaign  plan  from  the 
perspective  of  operational  assessment 


Recommend  to 
Strategy  Plans 
Team  for  branch 
and/or  sequel 
planning 
considerations 


*The  system  shall  aid  in  recommending  to 
Strategy  Plans  Team  for  branch  and/or 
sequel  planning  considerations 


AFOTTP 


Identify/establish 
linkages  with 
sources  of 
information  and 
intel  that  will  be 


*Know  who  to 
contact  for 
pieces  of 
information 


*Determine  critical 
sources  of  information 


*Determine  when 


*Establish  contacts 
with  resources 


*The  system  shall  provide  access  to 
various  types  of  resources 


*The  system  shall  provide  a  way  to  request 
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JAOP  Development  Consulting 


required  to 
support  JAOP 
development  in 
the  planning  stage 
and  a  dynamic 
Operational 
Assessment 
process  during 
execution  based 
on  information 
and  intelligence 
from  sources 
within  and  outside 
JAOC 

information  is  needed 

*Determine  reliable 
sources  of  information 

for  information 

Reg 

4 

Provide  a 
coherent  EBO 
representation 

*  Desired  effects 

*Linkage  to 
tasks/objectives 

*Rational 
behind  the 
decisions  made 
in  the  process 

*Determine  how  to  assess 
the  desired  effects  against 
objectives 

*Determine  how  to 
determine  if  objectives  are 
being  met 

*lndicators  are  a  good 
start  for  better  defining 
the  Commanders 
Crtiical  information 
requirements  (CCIR) 
and  providing 
increased  focus  for 

ISR  tasking 

*The  system  shall  aid  in  determining  how  to 
assess  the  desired  effects  against 
objectives 

*The  system  shall  aid  in  determining  how  to 
determine  if  objectives  are  being  met 
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*lndicators 
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JAOP  Development  Consulting 


*What  do  we  want  the 
enemy  to  do? 


*What  part  of  the  enemy 
must  we  effect  to  make 
them  do  it? 


*The  system  shall  aid  in  determining  what 
we  want  the  enemy  to  do? 


Reg 


Continue  linking 
5  COA  through  the 
MAAP 


*EBO  Lexicon 


*What  reaction  do  we 
anticipate  and  why  do  we 
think  this? 


*What  objectives  are  we 
supporting  and  how  does 
this  accomplish  them? 
*Why  do  we  want  to 
achieve  them  (effects  and 
objectives)? 


*Where  and  when  and  to 
what  degree  do  we  need 
to  attain  the  desired 
effects? 


Either  the  system  must 
provide  those 
executing  with  all  of 
the  relevant 
information  or  it  must 
provide  them  the 
means  with  which  to 
request  and  receive 
the  information  in  a 
timely  manner. 


*The  system  shall  aid  in  determining  what 
part  of  the  enemy  must  we  effect  to  make 
them  do  it? 


*The  system  shall  aid  in  determining  what 
reaction  do  we  anticipate  and  why  do  we 
think 


*The  system  shall  aid  in  determining  what 
objectives  are  we  supporting  and  how  does 
this  accomplish  them? 
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JAOP  Development  Consulting 


*lntegrate  multiple 
sources 


*The  system  shall  provide  the  capability  to 
integrate  multiple  sources 


*Blue  COMM, 
logistics,  space, 
operations,  10, 


*lntegrate  multiple 
collection  strategies 


MEBOPC 


Develop  an  ops 
assessment  plan 
for  JAOP  planning 
and  execution 


*ISR 


*enemy  intel 


*Commander 
preferred  option 
for  achieving 
the  end  state 


*Prior  intel 
collection  plans 
one  they  are 
available 


*Develop  procedures 
for  gathering  and 
exploiting 


*Determine  assessment 
shortfalls 


*ldentify  and  establish 
links  to  info/intel 
required  for  JAOP 
development 


*ldentify  and  establish 
linkages  to  info  or  intel 
sources  that  can 
support  CA  and  OA 


*Develop  and 
periodically 

review/refine  collection 
requirements  required 


*BDA 

*CA 


IBRD 


*MEA 


*MA 


*RR 


*The  system  shall  provide  the  capability  to 
integrate  multiple  collection  strategies 


*The  system  shall  provide  the  capability  to 
develop  procedures  for  gathering  and 
exploiting 


*The  system  shall  provide  the  capability  to 
identify  and  establish  links  to  info/intel 
required  for  JAOP  development 


*The  system  shall  provide  the  capability  to 
identify  and  establish  linkages  to  info  or 
intel  sources  that  can  support  CA  and  OA 


*The  system  shall  provide  the  capability  to 
develop  and  periodically  revew/refine 
collection  requirements  required  by  the 
blue  planners 
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JAOP  Development  Consulting 


by  the  blue  planners 


*ldentify  SI  for  00 


MEBOPC  7 


Develop  STTM 


*Determine/describe  the 
desired  change  to  those 
elements  and/or 
relationships  in  terms  of  a 
numerical  graphical  trend 


*Determine/describe  the 
threshold  of  change  to 
system  elements  and/or 
relationships  that 
indicates  completion  of 
the  related  action 


*ldentify  MOE  for  TO 


*ldentify  MOP  for  TT 


*Describe  the 
elements  and  or 
relationships  of  the 
system  that  need  to  be 
observed  in  order  to 
determine  whether  the 
assigned  actions  have 
been  completed 


'Component 

planners 


*The  system  shall  aid  in 
determining/describing  the  desired  change 
to  those  elements  and/or  relationships  in 
terms  of  a  numerical  graphical  trend 


*The  system  shall  aid  in 
determining/describing  the  threshold  of 
change  to  system  elements  and/or 
relationships  that  indicates  completion  of 
the  related  action 
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Monitor/Evaluate/Report  Air  Ops  Results  Related  to  JFACC  00 


Critical  Cue 
and/or 
Factors 


Critical  Decisions 


Actions 


Data 

Products 

used 


Comments 


Requirements 


Assess  direct 
and  indirect 
effects  of  air, 
space  and  10 
on  the 
established 
plan 


*Adversary 

behaviors 


*Mission 

successes/failures 


*Desired/undesired 

effects 


*Objectives 


*Changes  in 
environment  / 
weather 


*Determine  if  the  plan  was 
successful 


*Determine  if  measures 
were  effective 


*  Evaluate  data 


*The  system  shall  allow  and  aid  in  the 
assessment  of  both  direct  and  indirect 
effects  of  air,  space  and  10  on  the 
established  plan 


Derive 

intended  and 
unintended 
consequences 
of  air  ops  wrt 
platforms, 
munitions, 
culture, 
population 


*Adversary 

behaviors 


*Mission 

successes/failures 


*Desired/undesired 

effects 


*Objectives 


*Changes  in 
environment  / 
weather 


*Changes  in 
Adversary  behavior 


*Determine 

consequences 


*The  system  shall  allow  and  aid  in  the 
derivement  of  the  intended  and  unintended 
consequences  of  air  ops  wrt  platforms, 
munitions,  culture,  population 


*COAs 
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Determine 

*Plan  vs.  Actual 

*Mission  success 
vs  failures 

*Behind  plan 

when  and 

*Ahead  of  schedule 

what  to  report 
to  JFACC 

*Objectives 

*Not  going  to 
achieve  plan 

*Effects 

*Determine  the  difference 
between  the  plan  and 
actual  events 


*Determine  how  mission 
failures  will  affect  the 
overall  plan 


*Determine  how  effective 
mission  successes  were 


*Determine  if  we  are 
behind  plan 


*Determine  if  we  are 
ahead  of  schedule 


*Determine  if  current  COA 
will  effectively  achieve 
plan 


*Report  to 
JFACC 


Power 

Point 


*The  system  shall  allow  and  aid  in 
determining  when  and  what  to  report  to 
JFACC 


*The  system  shall  aid  in  determining  the 
difference  between  the  plan  and  actual 
events 


*The  system  shall  aid  in  determining  how 
mission  failures  will  affect  the  overall  plan 


*The  system  shall  aid  in  determining  how 
effective  mission  successes  were 


*The  system  shall  aid  in  determining  if  we 
are  behind  plan 


*The  system  shall  aid  in  determining  if  we 
are  ahead  of  schedule 


*The  system  shall  aid  in  determining  if 
current  COA  will  effectively  achieve  plan 


Report  on  air 
space  and  10 
execution 


*Plan  vs.  Actual 


*Mission  success 
vs  failures 


*Objectives 


*Behind  plan 


*Ahead  of  schedule 


*Not  going  to 
achieve  plan 


*Determine  what  to  report 
on 


*Determine  how  much 
information  to  put  in 
presentation 


Power 

Point 


*The  system  shall  aid  in  determining  what 
to  report  on 


*The  system  shall  aid  in  determining  how 
much  information  to  put  in  presentation 
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*Effects 


Recognize 
actionable 
changes  in 
ongoing  air 
ops 


*Unanticipated 
enemy  actions 


*current  Enemy 
COA 


*Change  in 
pathways 


*Change  in 
weather 


*Determine  when 
changes  have  been  made 
in  ongoing  air  ops 


*The  system  shall  allow  and  aid  in 
recognizing  actionable  changes  in  ongoing 
air  ops 


Monitor 

collection 

planning 


*Know  when,  who 
and  how  data  is 
going  to  be 
collected 


*Data  not  available 
on  deadline 


*Determine  if  another 
source  or  type  of  data  can 
be  used 


*Monitor 

Request  for 
more  other 
information 


ISRD 


*The  system  shall  keep  track  of  incoming 
data  according  to  the  when,  who  and  how 
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Synch 
collection 
planning  and 
execution 


*Mission 

successes/failures 


*Desired/undesired 

effects 


*Objectives 


*Mission 

successes/failures 


*DMPI  and  Sortie 
status 


*Actual  vs.  Planned 


*Determine  if  actual 
events  coincide  with  the 
plan 


*Determine  if  you  are 
achieving  your  plan 


*Determine  where  the 
plan  is  going  wrong 


*Determine  if  plan  needs 
to  be  modified 


*Evaluate  plan 
against  actual 
events 


*  Evaluate  data 


*The  system  shall  aid  in  determining  if 
actual  events  coincide  with  the  plan 


*The  system  shall  aid  in  determining  if  you 
are  achieving  your  plan 


*The  system  shall  aid  in  determining  where 
the  plan  is  going  wrong 


*The  system  shall  aid  in  determining  if  the 
plan  needs  to  be  modified 


Evaluate 

MOEs 


Collection  plan 


*Timing  of 
execution  of 
mission 


*Timing  when  data 
is  to  be  collected 
against  mission 


Compare  objectives  and 
tasks  to  MOEs  to 
determine  effectiveness 


*The  system  shall  provide  the  capability  to 
compare  objectives  and  tasks  to  MOEs  to 
determine  effectiveness 
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Evaluate 

MOPS 


*Collection  plan 


*Timing  of 
execution  of 
mission 


*Timing  when  data 
is  to  be  collected 
against  mission 


*Compare  tasks  to  MOPs 
to  determine  effectiveness 


*The  system  shall  provide  the  capability  to 
compare  tasks  to  MOPs  to  determine 
effectiveness 


Evaluate  Sis 


*Collection  plan 


*Timing  of 
execution  of 


*Timing  when  data 
is  to  be  collected 
against  mission 


*Compare  objectives  to 
Sis  to  determine 
effectiveness 


*The  system  shall  provide  the  capability  to 
compare  objectives  to  Sis  to  determine 
effectiveness 


Evaluate 

effects 


*Time  effect  is  to 
be  in  place 


*Anticipated 

*Unanticipated 


*Determine  if  we  actually 
implemented  effect 


*Determine  duration  of 
effect 


*The  system  shall  aid  in  determine  if  we 
actually  implemented  the  effect 


*The  system  shall  aid  in  determining  the 
duration  of  effect 
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Produce 

briefings 


*Recent  air  ops 


*Friendly  status 
brief 


*lntel  assessment 


*Options  and 
recommendations 


*JFACC  discussion 
topics 


*What  data  to 
include  in  report 


*Determine  what  to  put  in 
report 


*Determine 

recommendations 


*Determine  confidence 


*Determine  status 


*Performed 
twice  daily 
(Morning  and 
night) 


Power 

point 


*Recent  air  ops 


*Friendly  status 
brief 


*lntel  analysis 
assessment 


Components 

assessment 


*JFACC 

discussion 

topics 


*Provides  a 
baseline  for  JFC 
briefing 


*Provides 
accomplishment  of 
objectives 


*The  system  shall  provide  the  ability  to 
determine  what  to  put  in  report 


*The  system  shall  aid  in  determine 
recommendations 


*The  system  shall  aid  in  determining 
confidence 


*The  system  shall  aid  in  determining  status 


OAT  Chief 


■ 

Reference 

Code 

# 

CWR 

"IRR" 

Critical  Cue 
and/or  Factors 

Critical  Decisions 

Actions 

Common 

Errors 

Tools 

used 

Communicate 

with 

Data  Products 
used 

Requirements 
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OAT  Chief 

Reference 

Code 

# 

CWR 

"IRR" 

Critical  Cue 
and/or  Factors 

Critical  Decisions 

Actions 

Common 

Errors 

Tools 

used 

Communicate 

with 

Data  Products 
used 

( 

I 

Requirements 

I 

1 

AFOTTP 

1 

Set  up  daily 
meeting  to  discuss 
progress  on  air 
and  space 
objectives 

*Attend  meeting 

*The  system  shall 
provide  reminders  of  re¬ 
occurring  meetings 

AFOTTP 

2 

Run  daily  meeting 
to  discuss 
progress  on  air 
and  space 
objectives 

*Mission  for  the 
previous  day 

*Changes  in 
objectives 

*Success  of 
previous  day(s)  in 
regards  to 
objectives 

*Failures 

*  Effects 

*The  system  shall 
provide  a  means  to 
access  briefings  that 
were  briefed  during 
meeting 

AFOTTP 

3 

In  conjunction  with 
the  OAT  Chief  and 
the  CA  Team, 
make  re-attack 
recommendations 
based  on 
achievement  of 
operational  level 
objectives. 

*  Desired  effect 

*Undesired  effects 

*Unintended  effects 

*Adversary 

behaviors 

*Adversary 
behaviors  in 
reaction  to  mission 

*Success/failures 
of  missions 

*Determine  if  a  re¬ 
attack  is  needed 

*Make 

recommendation 

*Targeting 

Strategist 

*CA  Team 

*MEA 

*BDA 

*MA 

*The  system  shall 
provide  decision  aiding 
tools  to  determine  re¬ 
attack  needs 

AFOTTP 

4 

Execution  Floor 

OA 

Representative(s) 
Report  execution 
status  to  the  OAT 
Chief  and  make 
recommendations 
toward  the  current 
operational 
assessment. 

*Recommendations 
on  execution  status 

*Execution  status 
vs  planned  status 

*  Evaluate 
recommendations 

*Execution  Floor 

OA 

Representative(s) 

*The  system  shall 
provide  access  to 
Execution  Floor  OA 
Representative(s)  reports 

*The  system  shall 
provide  decision  aiding 
tools  to  compare  reports 
(actual  events)  to 
planned  events  to  help 
develop  a  new/modified 

OA  plan 
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OAT  Chief 

■ 

Reference 

Code 

# 

CWR 

"IRR" 

Critical  Cue 
and/or  Factors 

Critical  Decisions 

Actions 

Common 

Errors 

Tools 

used 

Communicate 

with 

Data  Products 
used 

Requirements 

AFOTTP 

5 

Battlefield 
Coordination 
Detachment  (BCD) 
Intel  Analyst  make 
recommendations 
to  the  OAT  Chief 
for  updates  to  the 
JFACC  operational 
assessment. 

*BCD  Intel 
recommendations 

*  Evaluate 
recommendations 

*Battlefield 
Coordination 
Detachment  (BCD) 
Intel  Analyst 

*The  system  shall 
provide  access  to 
Battlefield  Coordination 
Detachment  (BCD)  Intel 
Analyses  reports 

AFOTTP 

6 

Component 

Representatives 

make 

recommendations 
to  the  OAT  Chief 
for  updates  to  the 
JFACC  operational 
assessment. 

*Recommendations 
made  from 
component  reps 

*  Evaluate 
recommendations 

*Component 

Representatives 

*The  system  shall 
provide  access  to 
Component 

Representatives  reports 

*Positions  to  be 
filled 

AFOTTP 

7 

Identify  manning 
requirements. 
Work  to  source 
and  build  the  team. 

*Availability  of 
personnel 

*Oualifications  of 
personnel 

*Determine  how  to 
build  team 

*The  system  shall 
provide  team  information 
(availability,  role) 

*The  system  shall 
provide  a  way  to  assign 
tasks  to  team  members 

*Mission  to  be 
accomplished 

AFOTTP 

8 

Develop 
comprehensive 
assessment  plan, 
including  process 
assessment  plan 

*JFC/JFACC 

Objectives 

*Desired  effects 

*Success  of 
previous  day(s) 

*Failures 

*Determine  what 

OA  plan  will  be 

*Augment  staff  to 
develop  a  process 
assessment  plan 

*Not  have  all 
data  needed  to 
develop 
comprehensive 
plan 

*The  system  shall 
provide  decision  aiding 
tools  in  plan  development 

*Undesired  effects 

*  Effects 
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OAT  Chief 

■ 

Reference 

Code 

# 

CWR 

"IRR" 

Critical  Cue 
and/or  Factors 

Critical  Decisions 

Actions 

Common 

Errors 

Tools 

used 

Communicate 

with 

Data  Products 
used 

Requirements 

AFOTTP 

9 

Define  and 
document  roles 
and  responsibilities 
of  each  team 
member 

*What  roles  are 
and  their 
responsibilities 

Meet  with  each 
team  member  prior 
to  deployment 

*The  system  shall 
provide  the  ability  to 
define  and  document 
roles  and  responsibilities 
of  each  team  member 

AFOTTP 

1 

0 

Identify  specific 
people  to  fill 
positions 

*Use  AF  studies 
and  analysis 
agency  to  help 
identify  analysts  to 
meet  requirements 

*ldentify  process 
assessment  team. 
Bring  them  early 
into  the  planning 
process 

*The  system  shall 
provide  the  ability  to 
identify  specific  people  to 
fill  positions 

AFOTTP 

1 

1 

Determine 

clearance/access 

requirement 

*Team  Chief, 
Deputy  Team  Chief 
and  an  analyst 
need  COAL 
Warfighter  access 
(CW) 

*What  other 
positions  require 
what  clearances 

*The  system  shall 
provide  clearance/access 
requirement(s) 

AFOTTP 

1 

2 

Establish 
procedures  to 
ensure  the  OA 
team  published  a 
complete, 
accurate,  properly 
formatted,  and 
timely  OA  Report 

*Know  what  an 
accurate  report 
looks  like 

*Objectives 

*Task 

accomplishment 

*Plan 

*Determine  if  report 
is  accurate, 
complete,  and 
properly  formatted 

*Report  and 
assess  AOD,  ITO, 
Weaponeered 
Sourced  JPITL, 
MAAP  and  JAOP 
effectiveness  in 
terms  of  objective 
and  task 
accomplishment, 
adherence  or 
divergence  from 
the  established 
plan,  and  optimum 
use  of  available 
resources  (e.g. 
sorties,  munitions, 
etc.). 

*AOD 

*ITO 

*Weaponeered 
Sourced  JPITL 

*MAAP 

*JAOP 

*The  system  shall  supply 
a  standard  formatted 
report  to  use  and  to 
compare 
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OAT  Chief 

■ 

Reference 

Code 

# 

CWR 

"IRR" 

Critical  Cue 
and/or  Factors 

Critical  Decisions 

Actions 

Common 

Errors 

Tools 

used 

Communicate 

with 

Data  Products 
used 

Requirements 

AFOTTP 

1 

3 

Establish  standard 
operating 
procedures 
specific  to  the 
theater  of 
operations  (link  to 
SOP) 

*SPINS 

*Other  rules  of  the 

area 

*Objectives 

*The  system  shall  allow 
the  establishment  of 
standard  operating 
procedures  that  are 
specific  to  the  theater  of 
operations 

AFOTTP 

1 

4 

Establish  OAT 
battle  rhythm  and 
work  schedule 

*The  system  shall  allow 
the  establishment  of  an 
OAT  battle  rhythm  and 
work  schedule 

AFOTTP 

1 

5 

Participate  in  daily 
strategy  meetings 

*JFC/JFACC 

Objectives 

*Desired  effects 

*Undesired  effects 

*Determining  what 
is  measurable  and 
not  measurable  for 
the  OA  plan 

*Determining  if 
COA  can  be 
measured  and  are 
logical  for  the  OA 
team  to  assess  and 

measure 

*Participate  in 
meetings 

*Strat  team 

AFOTTP 

1 

6 

Brief  Operational 
Assessment  during 
the  daily  JFACC 
decision  briefing 

*Standard  format  of 
briefing 

*Color  coding 

*What  is  intended 
to  be  emphasized 

*Determine  color 
coding 

*Determine  critical 
pieces  of 
information 

*Produce  briefing 

*Take  too  long 
to  develop 
briefing 

*Not  have  up  to 
date  information 

*The  system  shall  easily 
transfer  data  into  a 
briefing  format 
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APPENDIX  B 


OEAVT  SYSTEM  MODEL  (CORE  EFFBDs) 


B-l 


B-2 


B-3 


B-4 


CORE  Database  for  Projcol;  OEAVT 


B-5 


B-6 


B-7 


B-8 


B-9 


B-10 


B-11 


B-12 


B-13 


B-14 


B-15 


B-16 


B-17 


B-18 


B-19 


B-20 


B-21 


B-22 


B-23 


B-24 


APPENDIX  C 


OEAVT  SYSTEM  REQUIREMENTS 


C-l 


Number  &  Name 

Description 

Status 

1.1  Meeting  Reminders 

The  system  shall  provide  reminders  of  re¬ 
occurring  meetings 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.2  Briefing  Access 

The  system  shall  provide  a  means  to  access 
briefings  that  were  briefed  during  meeting 

Accepted 

1.3  Re-attack  Decision 
Aiding  Tools 

The  system  shall  provide  decision  aiding 
tools  to  determine  re-attack  needs 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.4.1  Access  to  Execution 
Floor  OA  Reports 

The  system  shall  provide  access  to 

Execution  Floor  OA  Representative (s) 
reports 

16  Potential  Requirements 

1.4.2  Execution  Floor  OA 
Reports  Aiding  Tools 

The  system  shall  provide  decision  aiding 
tools  to  compare  reports  (actual  events)  to 
planned  events  to  help  develop  a 
new/modified  OA  plan 

Accepted 

1.5  BCD  Intel  Analysis 
access 

The  system  shall  provide  access  to 

Battlefield  Coordination  Detachment  (BCD) 
Intel  Analyses  reports 

Accepted 

1.6  Component 
Representative  Report 
access 

The  system  shall  provide  access  to 
Component  Representatives  reports 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.7.1  Team  Information 

The  system  shall  provide  team  information 
(availability,  role) 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.7.2  Team  Task 
Assignment 

The  system  shall  provide  a  way  to  assign 
tasks  to  team  members 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.8  Decision  Aiding  Tools 
for  Planning 

The  system  shall  provide  decision  aiding 
tools  in  plan  development 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.9  Team  Role 
Documentation 

The  system  shall  provide  the  ability  to 
define  and  document  roles  and 
responsibilities  of  each  team  member 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.10  Filling  Positions 

The  system  shall  provide  the  ability  to 
identify  specific  people  to  fill  positions 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.11  Clearance  Access 

The  system  shall  provide  clearance/access 
requirement(s) 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.12  Standard  Report 

The  system  shall  supply  a  standard 
formatted  report  to  use  and  to  compare 

Accepted 

1.13  SOP  procedures 

The  system  shall  allow  the  establishment  of 
standard  operating  procedures  that  are 
specific  to  the  theater  of  operations 

15  Out  of  Scope 

Spreadsheet  Requirements 

C-2 


Number  &  Name 

Description 

Status 

1.14  OAT  Battle  Rhythm 

The  system  shall  allow  the  establishment  of 
an  OAT  battle  rhythm  and  work  schedule 

15  Out  of  Scope 

Spreadsheet  Requirements 

1.15  Data  to  Briefing 

The  system  shall  easily  transfer  data  into  a 
briefing  format 

Accepted 

2.1.1  IPB  Assessment 

access 

The  system  shall  provide  access  to  IPB 
assessment 

Accepted 

2.1.2  Adversary  GOA 
access 

The  system  shall  provide  adversary  COAs 

16  Potential  Requirements 

2.1.3  COG  access 

The  system  shall  provide  access  to  COG 
information 

16  Potential  Requirements 

2.1.4  Targeting  Info  access 

The  system  shall  provide  targeting 
information 

21  Additional  Out-of-Scope 
Requirements 

2.1.5  ISROPs  access 

The  system  shall  provide  access  to  ISR 
operations 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.1.6  Adversary  Activity 
info 

The  system  shall  provide  information 
concerning  adversary  activity 

Accepted 

2.1.7  Adversary  Status 
info 

The  system  shall  provide  information  on 
adversary  status  in  battlespace 

Accepted 

2.1.8  ACF  Info  Access 

The  system  shall  access  to  ACF  information 

Accepted 

2.1.9  Adversary  Intention 
Info  Access 

The  system  shall  provide  adversary 
intentions 

16  Potential  Requirements 

2.1.10  Adversary  Strategy 
Info  Access 

The  system  shall  provide  adversary 
strategies 

16  Potential  Requirements 

2.1.11  Adversary  End 

State  Info  Access 

The  system  shall  provide  adversary  end 
state 

16  Potential  Requirements 

2.1.12  Cultural  Info  Access 

The  system  shall  provide  adversary 
"Cultural"  information 

16  Potential  Requirements 

2.2.1  CA  Access 

The  system  shall  provide  access  to  CA 

Accepted 

2.2.2  BDA  Access 

The  system  shall  provide  access  to  BDA 

Accepted 

2.2.3  MEA  Access 

The  system  shall  provide  access  to  MEA 

Accepted 

2.2.4  MA  Access 

The  system  shall  provide  access  to  MA 

Accepted 

2.2.5  OAT  &  Targeting 
Comms 

The  system  shall  provide  comms  between 
OAT  and  Targeting  specialist 

17  Defective  Originating 
Requirements 

C-3 


Number  &  Name 

Description 

Status 

2.2.6  Status  Updates 

The  system  shall  provide  updates  of  status 
ofCA,BDA,MEA,and  MA 

Accepted 

2.3.1  Access  to  JFC/JFACC 
Objectives  &  Effects 

The  system  shall  provide  access  to 

JFC/JFACC  objectives,  and  desired  effects 

Accepted 

2.3.2  Access  to  Targeting 
Strategy 

The  system  shall  provide  access  to 
targeting  strategy 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.3.3  Compare  Objectives 
&  Effects 

The  system  shall  provide  views  to  compare 
objectives,  and  desired  effects  to  targeting 
results 

19  Update  Requirements 
to  Reflect  Analysis 

2.4.1  Develop  Targeting 
Strategy 

The  system  shall  provide  the  capability  to 
develop  a  targeting  strategy 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.4.2  Promulgate  Plan  to 
Team 

The  system  shall  provide  a  means  to 
promulgate  plan  to  team 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.5  Compare  BDA/MEA  to 
MA 

The  system  shall  provide  the  capability  to 
compare  BDA/MEA  to  MA  to  assess 
operational  effectiveness 

19  Update  Requirements 
to  Reflect  Analysis 

2.6.1  Evaluate  BDA  for 
Effects 

The  system  shall  provide  the  capability  to 
evaluate  BDA  for  effects 

Accepted 

2.6.2  Access  to  Mission 
Outcome  data 

The  system  shall  provide  data  to  mission 
success,  failures,  shortcomings 

Accepted 

2.6.3  Access  to  Adversary 
data 

The  system  shall  provide  access  to 
adversary  history,  capabilities,  resources, 
and  intentions 

Accepted 

2.6.4  Team  BDA  Status 

The  system  shall  provide  ways  to  update 
team  on  status  of  BDA 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.7  Re-attack 
Recommendation 

The  system  shall  provide  means  to  make  a 
re-attack  recommendation 

Accepted 

2.8  TPED  Access 

The  system  shall  provide  access  to  TPED 

16  Potential  Requirements 

2.9.1  ISR  Strategy  Access 

The  system  shall  provide  access  to  ISR 
strategy 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.9.2  ISR  Strategy 
Modifications 

The  system  shall  allow  modifications  to  ISR 
strategy 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.10  Access  to  PIR/ISR 
Tasking 

The  system  shall  provide  access  to  PlR/lSR 
tasking 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.11  Evaluate  ISR  strategy 

The  system  shall  provide  a  means  to 
evaluate  ISR  strategy  by  evaluating  it 
against  PlRs,  strategies,  and  plans 

15  Out  of  Scope 

Spreadsheet  Requirements 

C-4 


Number  &  Name 
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Status 

2.12  Monitor  ISR 

Objectives 

The  system  shall  provide  the  ability  to 
monitor  ISR  objectives 

15  Out  of  Scope 

Spreadsheet  Requirements 

2.13  ISR  Report  Access 

The  system  shall  provide  access  to  ISR 
report 

Accepted 

2.14  ISR  Documentation 
Access 

The  system  shall  provide  access  to  ISR 
documentation 

16  Potential  Requirements 

2.15  Access  Multiple 
sources  of  Reports 

The  system  shall  provide  access  to  multiple 
sources  of  reports 

Accepted 

2.16  Evaluation  of  PIR 

The  system  shall  allow  the  tracking  of  PIR 
status 

19  Update  Requirements 
to  Reflect  Analysis 

2.17  Evaluation  of  MOEs 

The  system  shall  allow  evaluation  of  MOEs 

Accepted 

2.18  Evaluation  of  Intel 

Data 

The  system  shall  allow  evaluation  and 
comparison  of  summarized  intel  data  with 
other  data 

22  Duplicate  System 
Requirements 

3.1  Aid  Actual  Action 
Evaluation 

The  system  shall  aid  in  tracking  actions 
against  the  plan 

19  Update  Requirements 
to  Reflect  Analysis 

3.2  Environment  Affects 
Plan 

The  system  shall  aid  in  determining  the 
potential  effect  of  weather,  terrain,  and/or 
air  spaces  current/future  on  possible  plan 
changes 

18  Rewording  of 
Requirements 

3.3  Determine  if  Tactical 
Objectives  Achieved 

The  system  shall  aid  in  determining  if 
tactical  objectives  will  be  achieved 

Accepted 

3.4  Determine  if 

Operational  Objectives 
Achieved 

The  system  shall  aid  in  determining  if 
operational  objectives  will  be  achieved 

Accepted 

4.1  Assess  Effects  on  Plan 

The  system  shall  allow  and  aid  in  the 
assessment  of  both  direct  and  indirect 
effects  of  air,  space  and  10  on  the 
established  plan 

Accepted 

4.2  Derive  Consequences 

The  system  shall  allow  and  aid  in  the 
derivation  of  the  intended  and  unintended 
consequences  of  air  ops  wrt  platforms, 
munitions,  culture,  population 

Accepted 

4.3.1  JFACC  Report  Aids 

The  system  shall  allow  and  aid  in 
determining  when  and  what  to  report  to 
JFACC 

Accepted 

4.3.2  Aid  determining  Plan 
vs  Actual 

The  system  shall  aid  in  identifying 
deviations  from  the  plan 

19  Update  Requirements 
to  Reflect  Analysis 

C-5 


Number  &  Name 
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Status 

4.3.3  Evaluate  Mission 
Failures 

The  system  shall  aid  in  determining  how 
mission  failures  will  affect  the  overall  plan 

Accepted 

4.3.4  Evaluate  Mission 
Successes 

The  system  shall  aid  in  determining  the 
contributions  of  individual  actions  to 
operational  effects 

19  Update  Requirements 
to  Reflect  Analysis 

4.3.5  Determine  if  behind 
plan 

The  system  shall  aid  in  determining  if  we 
are  behind  plan 

Accepted 

4.3.6  Determine  if  Ahead 
Schedule 

The  system  shall  aid  in  determining  if  we 
are  ahead  of  schedule 

Accepted 

4.3.7  Determine  if  GOA 
effective 

The  system  shall  aid  in  determining  if  the 
COA  is  achieving  objectives 

18  Rewording  of 
Requirements,  19  Update 
Requirements  to  Reflect 
Analysis 

4.4.1  Aid  in  Report  subject 

The  system  shall  aid  in  determining  what 
aspects  of  the  mission  should  be  reported 
in  the  daily  briefing 

21  Additional  Out-of-Scope 
Requirements 

4.4.2  Aid  in  Report 
contents 

The  system  shall  aid  in  determining  how 
much  information  to  put  in  presentation 

17  Defective  Originating 
Requirements 

4.5  Recognize  Actionable 
Changes 

The  system  shall  allow  and  aid  in  detecting 
actionable  changes  in  ongoing  air  ops 

19  Update  Requirements 
to  Reflect  Analysis 

4.6  Track  Incoming  Data 

The  system  shall  keep  track  of  incoming 
data  according  to  the  when,  who  and  how 

Accepted 

4.7.1  Actual  Events  vs 

Plan 

The  system  shall  aid  in  determining  if 
actual  events  coincide  with  the  plan 

Accepted 

4.7.2  Determine  if  Plan 
Achieved 

The  system  shall  aid  in  determining  if  you 
are  achieving  your  plan 

Accepted 

4.7.3  Determine  if  Plan 
going  wrong 

The  system  shall  aid  in  determining  where 
the  plan  is  going  wrong 

Accepted 

4.7.4  Determine  if  Plan 
needs  mods 

The  system  shall  aid  in  determining  if  the 
plan  needs  to  be  modified 

Accepted 

4.8  Compare 
Objectives/Tasks  to  MOEs 

The  system  shall  provide  the  capability  to 
compare  objectives  and  tasks  to  MOEs  to 
determine  effectiveness 

Accepted 

4.9  Compare  Tasks  to 

MOPs 

The  system  shall  provide  the  capability  to 
compare  tasks  to  MOPs  to  determine 
effectiveness 

Accepted 

C-6 


Number  &  Name 
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4.10  Compare  Objectives 
to  Sis 

The  system  shall  provide  the  capability  to 
compare  objectives  to  Sis  to  determine 
effectiveness 

Accepted 

4.11.1  Determine  if  effect 
implemented 

The  system  shall  aid  in  determine  if  we 
actually  achieved  the  effect 

18  Rewording  of 
Requirements 

4.11.2  Determine 
persistence  of  effect 

The  system  shall  aid  in  determining  the 
persistence  of  the  effect 

16  Potential  Requirements, 
18  Rewording  of 
Requirements 

4.12.1  Determine  Report 
Contents 

The  system  shall  provide  the  ability  to 
determine  what  to  put  in  report 

Accepted 

4.12.2  Aid  with 
Recommendations 

The  system  shall  aid  in  determine 
recommendations 

Accepted 

4.12.3  Aid  with 

Confidence 

The  system  shall  aid  in  determining 
confidence 

Accepted 

4.12.4  Aid  with  Status 

The  system  shall  aid  in  determining  status 

Accepted 

5.1  Develop  Strat  Plan 
options 

The  system  shall  provide  a  means  to 
develop  options  to  strat  plans  and  compare 
courses 

15  Out  of  Scope 

Spreadsheet  Requirements 

5.2  Compare  TT  actual  vs 
expected 

The  system  shall  allow  the  comparison  of 
actual  to  expected  results  for  tactical  tasks 

Accepted 

5.3.1  Identify  shortfalls  in 
plans 

The  system  shall  identify  shortfalls  in 
missions  and  plans 

Accepted 

5.3.2  Overcoming  shortfall 
decision  aiding 

The  system  shall  support  the  decision 
making  of  overcoming  the  shortfalls 

Accepted 

5.3.3  Plan  Modification 
decision  aiding 

The  system  shall  aid  in  determining  if  a 
modification  to  the  plan  is  needed 

Accepted 

5.4.1  Corrective  Action 
Feasibility  Assessment 
support 

The  system  shall  aid  in  and  allow  the 
assessment  of  the  feasibility  of  the 
corrective  actions 

Accepted 

5.4.2  Corrective  Action 
resource  analysis 

The  system  shall  determine  if  resources  are 
available  for  corrective  actions 

15  Out  of  Scope 

Spreadsheet  Requirements 

5.4.3  Plan  Change  analysis 

The  system  shall  determine  what  to  change 

18  Rewording  of 
Requirements 

5.4.4  WOE  Analysis 

The  system  shall  support  WOE  analysis 
and  modeling 

19  Update  Requirements 
to  Reflect  Analysis 

5.4.5  Objective  Time 
Analysis 

The  system  shall  determine  the  time 
needed  to  achieve  each  objective 

Accepted 

C-7 
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5.4.6  Determine  Plan 
priorities 

The  system  shall  determine  the  priorities 
for  next  day  according  to  actual  vs  planned 

15  Out  of  Scope 

Spreadsheet  Requirements 

5.5.1  Predict  Corrective 
Action  Effects 

The  system  shall  allow  and  aid  in  the 
prediction  of  effects  of  the  corrective 
actions  on  tactical  and  operational 
objectives 

Accepted 

5.5.2  Predict  Corrective 
Action  Effects  on  TO  and 

00s 

The  system  shall  allow  and  aid  in  the 
prediction  of  the  effects  that  the  corrective 
actions  on  tactical  and  operational 
objectives  will  have 

16  Potential  Requirements 

5.6.1  Determine  Best  Plan 
for  Objectives 

The  system  shall  determine  which  plan  will 
"best"  accomplish  objectives/effects 

15  Out  of  Scope 

Spreadsheet  Requirements 

5.6.2  Determine  Best  Plan 
for  Risks 

The  system  shall  determine  which  plan  has 
less  risk  associated  with  it 

15  Out  of  Scope 

Spreadsheet  Requirements 

5.7.1  Select  Corrective 
Actions 

The  system  shall  allow  and  aid  in  the 
selection  of  one  or  more  corrective  actions 

21  Additional  Out-of-Scope 
Requirements 

5.7.2  Provide  Report 
Templates 

The  system  shall  provide  a  template  for 
reports 

Accepted 

5.7.3  Aid  in 

Communications  to  Plans 
Team 

The  system  shall  allow  and  aid  in  the 
communication  of  the  corrective  actions  to 
the  plans  team 

Accepted 

5.8.1  Overall  Effect 
limitations 

The  system  shall  aid  in  determining  what 
limitations  will  have  on  achieving  overall 
effect 

15  Out  of  Scope 

Spreadsheet  Requirements 

5.8.2  Overcome 

Limitations 

The  system  shall  aid  in  determining  how  to 
overcome  limitations 

15  Out  of  Scope 

Spreadsheet  Requirements 

6.1.1  Receive  CA  Info 

The  system  shall  provide  a  way  to  receive 
combat  assessment  information 

Accepted 

6.1.2  Request  CA  Info 

The  system  shall  provide  a  way  to  request 
information 

Accepted 

6.1.3  New  Report  Alert 

The  system  shall  provide  a  way  to  "alert" 
people  that  a  new  report  is  available 

Accepted 

6.2  Aid  Correlating  CA 
Inputs 

The  system  shall  aid  in  correlating  CA 
inputs  to  the  mission 

Accepted 

6.3.1  Access  to  ISR  data 

The  system  shall  provide  access  to  ISR  data 
(summaries) 

Accepted 

C-8 


Number  &  Name 
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Status 

6.3.2  Rate  ISR  Data 

The  system  shall  provide  a  way  to  compare 
data  and  rate  its  validity,  reliability, 
credibility,  and  efficacy. 

19  Update  Requirements 
to  Reflect  Analysis 

6.4.1  Correlate  CA  data 

The  system  shall  aid  in  correlating  CA 
results  with  TT,  TO,  and  00 

Accepted 

6.4.2  Determine  if 
Objectives  met 

The  system  shall  aid  in  determining  if 
objectives  are  being  met 

Accepted 

6.4.3  Aid  determining 

WOE  status 

The  system  shall  aid  in  determining  if  WOE 
should  be  maintained  or  changed 

Accepted 

6.4.4  Determine  data 
Correlation/Contradiction 

The  system  shall  allow  evaluation  and 
comparison  of  all  summarized  operational 
data. 

19  Update  Requirements 
to  Reflect  Analysis 

6.4.5  Determine  if 
priorities  maintained 

The  system  shall  display  the  priority 
associated  with  each  objective. 

19  Update  Requirements 
to  Reflect  Analysis 

6.4.6  Determine  if  plan 
mods  needed 

The  system  shall  aid  in  determining  if  a 
modification  to  the  plan  is  needed 

Accepted 

6.5.1  Display  objective 
progress 

The  system  shall  display  progress  towards 
objectives 

Accepted 

6.5.2  Display  Plan 

The  system  shall  display  the  Plan 

15  Out  of  Scope 

Spreadsheet  Requirements 

6.5.3  Display  WOE 

The  system  shall  display  the  WOE 

Accepted 

6.5.4  Display  Mission 

Time 

The  system  shall  display  the  mission  Time 
(day  and  hours) 

18  Rewording  of 
Requirements 

6.5.5  Display  Plan  vs 

Actual  Tasking 

The  system  shall  display  planned  versus 
actual  tasking 

18  Rewording  of 
Requirements 

6.5.6  Aid  is  Objective 
progress  determination 

The  system  shall  aid  in  determining  if 
objectives  are  being  met 

Accepted 

6.5.7  Aid  in  WOE  status 

The  system  shall  aid  in  determining  if  WOE 
should  be  maintained  or  changed 

Accepted 

6.5.8  Aid  in  Plan  Mods 

The  system  shall  aid  in  determining  if  a 
modification  to  the  plan  is  needed 

Accepted 

6.5.9  Aid  determining  Plan 
Progress 

The  system  shall  aid  in  determining  if  you 
are  ahead/behind  plan 

Accepted 

6.5.10  Aid  determining 

Plan  requires  additional 
time 

The  system  shall  aid  in  determining  if 
additional  time  is  needed  to  achieve 
operational  objectives 

19  Update  Requirements 
to  Reflect  Analysis 

C-9 


Number  &  Name 

Description 

Status 

6.6.1  Derive  intended  & 
unintended  effects  from 

CA 

The  system  shall  provide  a  way  to  derive 
unintended  and  intended  effects  from  CA 
results 

Accepted 

6.6.2  Aid  in  Effects 

Analysis 

The  system  shall  aid  in  determining  if 
effects  are  being  achieved 

Accepted 

6.6.3  Determine  cause  of 
effect 

The  system  shall  aid  in  determining  what  is 
causing  the  effect 

Accepted 

6.6.4  Aid  in  Determining 
duration  of  effect 

The  system  shall  aid  in  determining  how 
the  effect  is  being  achieved  and  how  long  it 
will  last 

17  Defective  Originating 
Requirements 

6.7.1  Monitor  CA  Channels 

The  system  shall  provide  a  capability  to 
monitor  CA  channels 

15  Out  of  Scope 

Spreadsheet  Requirements 

6.7.2  Organize  Data 

The  system  shall  organize  data 

17  Defective  Originating 
Requirements 

7.1  Develop  Assessment 
Measures 

The  system  shall  support  in  developing 
assessment  measures  for  missions 

Accepted 

7.2.1  Predict  COA  Events 

The  system  shall  aid  in  the  prediction  of 
events  that  may  occur  during  a  COA  (play 
out) 

15  Out  of  Scope 

Spreadsheet  Requirements 

7.2.2  Graphical 
Representation  of 
Battlespace 

The  system  shall  provide  a  graphical 
representation  of  the  Battlespace 

18  Rewording  of 
Requirements 

7.2.3  Access  Adversary 
data 

The  system  shall  have  access  to  adversary 
data 

Accepted 

7.3  Aid  in  COA 
development  to  achieve 
end  state 

The  system  shall  provide  a  means  to  input 
end  states  and  aid  in  the  development  of 
COAS  to  achieve  end  state 

15  Out  of  Scope 

Spreadsheet  Requirements 

7.4  Enter  Goal  Rationale 

The  system  shall  provide  access  to  or  a  way 
to  input  the  purpose  or  rational  as  to  why 
the  goal  is  sought 

15  Out  of  Scope 

Spreadsheet  Requirements 

7.5.1  Visualize 
Goal/Objective  Actions 

The  system  shall  provide  visually,  a  plan  or 
sequence  of  actions  on  how  the  goal  or 
objective  is  going  to  be  accomplished 

15  Out  of  Scope 

Spreadsheet  Requirements 

7.5.2  Access 

Environmental  data 

The  system  shall  provide  access  to 
environmental  information  such  as 
weather. 

Accepted 

C-10 
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7.6  Display  Available 
Resources 

The  system  shall  provide  access  to 
documents  and  resources  that  support  the 
plan. 

19  Update  Requirements 
to  Reflect  Analysis,  21 
Additional  Out-of-Scope 
Requirements 

7.7.1  Aid  in  OA  info 
determination 

The  system  shall  aid  in  determining  what 
info  is  required  to  support  OA 

20  Additional  Defective 
Requirements 

7.7.2  Allow  User  Requests 

The  system  shall  provide  the  capability  to 
generate  RFl's. 

19  Update  Requirements 
to  Reflect  Analysis,  23 
Requirements  downgraded 
from  mandatory  to 
potential  Requirements 

8.1.1  Access  Mission 
Guidance 

The  system  shall  provide  access  to  mission 
guidance 

18  Rewording  of 
Requirements 

8.1.2  List  Team  Members 

The  system  shall  provide  a  list  of  team 
members 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.1.3  Contact  Team 
Members 

The  system  shall  provide  a  means  to 
contact  team  members 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.2.1  Communication 

Paths 

The  system  shall  provide  various 
communication  pathways 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.2.2  TestComms 

The  system  shall  provide  ways  to  test 
comms 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.2.3  Diagnose  Comms 

The  system  shall  provide  a  diagnosis  of 
comms 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.2.4  Aid  in  Choice  of 
comm  mode 

The  system  shall  aid  in  the  decision  making 
of  choosing  a  comm  mode  for  mission 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.2.5  Display  Comm  Paths 

The  system  shall  display  various 
communication  pathways 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.3.1  TestComms 
(duplicate?) 

The  system  shall  provide  methods  to  test 
comms 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.3.2  Diagnose 
Comms(duplicate?) 

The  system  shall  provide  a  diagnosis  of 
problems  with  comms 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.3.3  Problem  Checklists 

The  system  shall  provide  checklists  on  how 
to  fix  problem 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.4.1  Provide  on-line 

users 

The  system  shall  provide  status  of  who  is 
on-line 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.4.2  Provide  team 
summary 

The  system  shall  provide  a  team  summary 

15  Out  of  Scope 

Spreadsheet  Requirements 

Number  &  Name 

Description 

Status 

8.5  List  comm  modes 

The  system  shall  provide  listings  of  various 
comm  modes 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.6.1  Set  up  Comm 
pathways 

The  system  shall  enable  users  to  set  up 
comm  pathways 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.6.2  Pathway  Alerts 

The  system  shall  allow  a  alerts  to  let  team 
members  know  what  the  comms  pathway 
being  used  is 

15  Out  of  Scope 

Spreadsheet  Requirements 

8.7  Database  Storage  of 
Information 

The  system  shall  allow  storage  of 
information  in  a  database 

Accepted 

9.1.1  JAOP  supports  Plan 

The  system  shall  aid  in  determining  how 
effectively/thoroughly  JAOP  supports 
campaign  plan 

15  Out  of  Scope 

Spreadsheet  Requirements 

9.1.2  JAOP  supports  OA 

The  system  shall  aid  in  determining  how 
thoroughly  and  effectively  it  supports  the 
overall  theater  campaign  plan  from  the 
perspective  of  operational  assessment 

17  Defective  Originating 
Requirements 

9.2  Branch/Sequel 
Recommendations 

The  system  shall  aid  in  recommending  to 
Strategy  Plans  Team  for  branch  and/or 
sequel  planning  considerations 

Accepted 

9.3.1  Resource  Access 

The  system  shall  provide  access  to  various 
types  of  resources 

17  Defective  Originating 
Requirements 

9.3.2  Information  Request 
support 

The  system  shall  provide  a  way  to  request 
for  information 

17  Defective  Originating 
Requirements 

9.4.1  Aid  Effects 

Assessment 

The  system  shall  aid  in  determining  how  to 
define  desired  effects  for  each  objective. 

19  Update  Requirements 
to  Reflect  Analysis,  23 
Requirements  downgraded 
from  mandatory  to 
potential  Requirements 

9.4.2  Aid  Determining 
Objectives  are  Met 

The  system  shall  aid  in  determining  if 
objectives  are  being  met 

18  Rewording  of 
Requirements 

9.5.1  Integrate  Multiple 
Sources 

The  system  shall  provide  the  capability  to 
integrate  multiple  sources 

17  Defective  Originating 
Requirements 

9.5.2  Integrate  Multiple 
Collection  Strategies 

The  system  shall  provide  the  capability  to 
integrate  multiple  collection  strategies 

15  Out  of  Scope 

Spreadsheet  Requirements 

9.5.3  Develop  Gathering  & 
Exploiting  Procs 

The  system  shall  provide  the  capability  to 
develop  procedures  for  gathering  and 
exploiting 

15  Out  of  Scope 

Spreadsheet  Requirements 

C-12 


Number  &  Name 

Description 

Status 

9.5.4  Intel  Links  for  JAOP 
Development 

The  system  shall  provide  the  capability  to 
identify  and  establish  links  to  info/intel 
required  for  JAOP  development 

15  Out  of  Scope 

Spreadsheet  Requirements 

9.5.5  Intel  Links  for  CA  & 

OA 

The  system  shall  provide  the  capability  to 
identify  and  establish  linkages  to  info  or 
intel  sources  that  can  support  CA  and  OA 

Accepted 

9.5.6  Develop  Collection 
Requirements 

The  system  shall  provide  the  capability  to 
develop  and  periodically  review/refine 
collection  requirements  required  by  the 
blue  planners 

15  Out  of  Scope 

Spreadsheet  Requirements 

9.6.1  Formulate  Effects 
Indicators 

The  system  shall  aid  in  formulating 

Sis,MOEs,  MOPs  and  Eis  for  desired  effects. 

18  Rewording  of 
Requirements,  19  Update 
Requirements  to  Reflect 
Analysis,  21  Additional 
Out-of-Scope 

Requirements 

9.6.2  Action  Completion 
Threshold  Determination 

The  system  shall  aid  in 
determining/describing  the  threshold  of 
change  to  system  elements  and/or 
relationships  that  indicates  completion  of 
the  related  action 

Accepted 

10.1.1  Develop  OA  Plan 

The  system  shall  provide  the  ability  to 
develop  an  OA  plan 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.1.2  Compare  OA  to 

JAOP 

The  system  shall  provide  the  ability  to 
compare  and  evaluate  the  OA  plan  against 
the  JAOP 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.1.3  Perform  OA  during 
execution 

The  system  shall  provide  the  ability  to 
perform  OA  in  real  time  during  execution 

18  Rewording  of 
Requirements 

10.1.4  Mod  OA  during 
execution 

The  system  shall  provide  the  ability  to 
modify  the  OA  plan  in  real  time 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.1.5  Aid  in  Decision 
Making 

The  system  shall  aid  in  the  decision  making 
of  where  and  how  the  plan  should  be 
modified 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.2  Develop  Team 
member  list 

The  system  shall  provide  the  capability  to 
develop  team  member  list(s) 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.3  Create  Assessment 
Plan 

The  system  shall  allow  for  the  creation  of 
an  assessment  plan 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.4  Integrate  JFC&JFACC 

The  system  shall  support  the  integration  of 
both  JFC  and  JFACC  guidance  and  feedback 

18  Rewording  of 
Requirements 

C-13 


Number  &  Name 

Description 

Status 

10.5  Support  Effectiveness 
&  Efficiency  Evaluation 

The  system  shall  support  the  evaluation  of 
air,  space,  and  10  effectiveness  and 
efficiency 

18  Rewording  of 
Requirements 

10.6.1  Share  Info  with 

Team 

The  system  shall  support  the  capability  to 
share  information  among  team  member 

19  Update  Requirements 
to  Reflect  Analysis 

10.6.2  Integrate  member 
data  in  OA  Plan 

The  system  shall  allow  and  support  the 
capability  to  integrate  data  received  from 
team  members  into  the  overall  OA  plan 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.7  Develop  Assessment 
Requirements 

The  system  shall  support  and  allow  the 
development  of  assessment  information 
requirements 

Accepted 

10.8.1  Evaluate  JFACC 
Objectives 

The  system  shall  support  evaluation  of 

JFACC  objectives 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.8.2  Provide  plan  access 

The  system  shall  allow  access  to  JAOP, 

MAA,  ATO,  JIPTL,  and  OPLAN 

Accepted 

10.8.3  Provide  plan 
request 

The  system  shall  allow  the  ability  to 
request  the:  JAOP,  MAA,  ATO,  JIPTL,  and 
OPLAN 

17  Defective  Originating 
Requirements 

10.8.4  Aid  decision 
between  product  & 
Objectives 

The  system  shall  aid  in  the  decision  making 
and  evaluation  between  mission  results 
and  objectives 

18  Rewording  of 
Requirements 

10.9  Support  Intel  source 
Links 

The  system  shall  support  and  aid  in  the 
identification  and  establishment  of  any 
definitive  linkages  to  information  or  intel 
sources  that  can  support  combat  and 
Operational  Assessment  functions  within 
the  JAOC 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.10.1  Refine  SOP 

The  system  shall  allow  for  the  refinement 
of  standard  operation  procedures  specific 
to  individual  duties  and  responsibilities 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.10.2  Assign  SOP  to 
individuals 

The  system  shall  allow  assignment  of  SOP 
to  individuals 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.10.3  Assign 
Responsibilities  to 
individuals 

The  system  shall  allow  assignment  of  roles 
and  responsibilities  to  individuals 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.10.4  ID  SOPs  and  roles 

The  system  shall  aid  in  the  identification  of 
SOPs,  roles,  and  responsibilities 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.11  Determine  Info 
requirements 

The  system  shall  support  determination  of 
information  requirements  associated  with 
assessments 

19  Update  Requirements 
to  Reflect  Analysis 

C-14 


Number  &  Name 

Description 

Status 

10.12  Request  Intel  Info 

The  system  shall  allow  support  obtaining 
Intel  and  other  operational  results. 

19  Update  Requirements 
to  Reflect  Analysis 

10.13.1  Integrate  JFACC 

OA  with  JFC 

The  system  shall  allow  the  integration  of 
the  OA  from  JFACC  with  its  counterparts  at 
the  JFC  level 

17  Defective  Originating 
Requirements 

10.13.2  Aid  in  OA  Plan 
Assessment 

The  system  shall  aid  in  the  assessment  of 
the  OA  Plan  to  ensure  a  cohesive  picture 
between  the  campaign  plan  and  the  air  and 
space  portion  of  that  campaign 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.14.1  Aid  in 
determining  loss 

The  system  shall  aid  and  support  in 
determining  loss 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.14.2  Aid  in 
determining  cost 

The  system  shall  aid  and  support  in 
determining  cost 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.14.3  Aid  in 
determining  if  COA  routine 

The  system  shall  aid  and  support  in 
determining  if  COA  is  routine  or  not 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.14.4  Aid  in 
determining  risk  levels 

The  system  shall  aid  and  support  in 
ensuring  that  standards  for  routine  events 
are  adequate  to  provide  an  acceptable  level 
of  risk 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.15.1  Aid  in  balancing 
risk  &  benefits 

The  system  shall  allow  and  aid  in  the 
balancing  of  risk  vs  benefits 

21  Additional  Out-of-Scope 
Requirements 

10.15.2  Aid  in  eliminating 
risks 

The  system  shall  aid  in  identifying 
operational  risks. 

18  Rewording  of 
Requirements,  21 

Additional  Out-of-Scope 
Requirements 

10.15.3  Aid  in  reducing 
magnitude  of  risks 

The  system  shall  aid  in  an  allow  to  reduce 
the  magnitude  of  mission  essential  risks  by 
applying  controls 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.16.1  Evaluate  roles  in 
EBA  process 

The  system  shall  allow  for  the  evaluation  of 
roles  and  responsibilities  of  components, 
coalition  members,  and  the  DIE  agencies  in 
the  EBA  process 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.16.2  Evaluate 
intelligence  requirements 

The  system  shall  allow  for  the  evaluation  of 
intelligence  collection  requirements 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.16.3  Evaluate  Battle 
Rhythm  for  MOE  &  MP 

The  system  shall  allow  for  the  evaluation  of 
battle  rhythm  to  track  and  continuously 
review  the  MOE  and  MP 

15  Out  of  Scope 

Spreadsheet  Requirements 

10.16.4  Plan 
Troubleshooting 

The  system  shall  support  plan 
troubleshooting 

18  Rewording  of 
Requirements 

C-15 


Number  &  Name 

Description 

Status 

10.16.5  Evaluate 
Methodology  for  Lack  of 
Progress 

The  system  shall  for  the  evaluation  of  the 
methodology  to  determine  whether  lack  of 
effects  achievement  is  due  to  inappropriate 
actions  or  monitoring  wrong  MOE 

17  Defective  Originating 
Requirements 

11.1.1  Provide  actions  for 
Goal 

The  system  shall  provide  a  plan  or 
sequence  of  actions  needed  to  achieve  the 
goal 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.1.2  Modify  Plan 

The  system  shall  provide  a  way  to  modify 
plan 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.2.1  Analyze  friendly 
COGs 

The  system  shall  provide  the  ability  to 
analyze  and  identify  friendly  COGs 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.2.2  ID  COG  capabilities 

The  system  shall  provide  the  ability  to 
identify  COG  Critical  capabilities 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.2.3  ID  COG 
requirements 

The  system  shall  provide  the  ability  to 
identify  COG  Critical  requirements 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.2.4  ID  COG 
vulnerabilities 

The  system  shall  provide  the  ability  to 
identify  COG  Critical  Vulnerabilities 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.3.1  Assess  Likelihood 
desired  Effects 

The  system  shall  provide  the  capability  to 
assess  the  likelihood  of  the  desired  effects 
attaining  the  objective 

Accepted 

11.3.2  Recommend 

Success  Indicators 

The  system  shall  provide  the  capability  to 
recommend  success  indicators  (MOEs)  that 
will  aid  in  assessment  of  desired  effects 
(ISR  Strategy  and  planning — third  pillar  of 
PBA) 

Accepted 

11.3.3  Determine 
unintended  Effects 

The  system  shall  provide  the  capability  to 
Determine  potential  unintended  effects 

Accepted 

11.3.4  Determine  Impact 
of  unintended  Effects 

Assess  the  impact,  or  value,  of  the 
unintended  effects  with  respect  to  the 
JFACC's  and  the  JFC's  objectives 

Accepted 

11.3.5  Unintended  Effects 
Indicators 

Recommend  indicators  that  will  aid  in 
assessment  of  unintended  effects  (ISR 
Strategy  and  planning — third  pillar  of  PBA) 

Accepted 

11.3.6  Determine  TO  for 
00s 

The  system  shall  provide  the  capability  to 
Determine  the  tactical  objectives  that  will 
accomplish  operational  objectives. 

Determine  desired  effects  (direct  and 
indirect)  that  will  achieve  objectives 

15  Out  of  Scope 

Spreadsheet  Requirements 

C-16 


Number  &  Name 

Description 

Status 

11.3.7  Determine 
likelihood  attaining 
objectives 

The  system  shah  provide  the  capability  to 
Assess  the  likelihood  of  the  desired  effects 
attaining  the  objective.  This  requires  an 
understanding  of  the  cause  and  effect 
relationship  (causal  linkage) 

22  Duplicate  System 
Requirements 

11.3.8  Determine 
Supporting  Actions/Tasks 

The  system  shah  provide  the  capability  to 
Determine  supporting  actions/tasks 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.3.9  Refine  COAs 

The  system  shah  provide  the  capability  to 
Refine  COAs  based  on  priority,  sequence, 
phasing,  weight  of  effort,  and  matched 
resources 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.3.10  Ensure  phases 
support  JFC 

The  system  shah  provide  the  capability  to 
Ensure  Phases  (and  objectives)  support  JFC 
phasing,  objectives,  and  end  state 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.3.11  Sequence  Tasks 

The  system  shah  provide  the  capability  to 
Sequence  Tasks  (Actions)  to  accomplish 
objectives  and  to 

15  Out  of  Scope 

Spreadsheet  Requirements 

11.3.12  ID  GOA  risks 

The  system  shah  provide  the  capability  to 
Identify  risk  areas  for  each  GOA 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.1  Access  Guidance  & 
Objectives 

The  system  shall  allow  access  to  guidance 
and  objectives 

Accepted 

12.2  Develop  Guidance  & 
Objectives 

The  system  shall  provide  the  ability  to 
develop  guidance  and  objectives 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.3.1  Generate  TOS 

The  system  shah  allow  generation  of  Tos 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.3.2  Provide  TOS 
Template 

The  system  shall  provide  a  template  to 
develop  Tos 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.3.3  Save  Objectives 

The  system  shah  provide  a  way  to  save  the 
objectives 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.4.1  Generate  MOEs 

The  system  shall  allow  generation  of  MOEs 

Accepted 

12.4.2  Save  MOEs 

The  system  shall  provide  a  way  to  save 

MOEs 

Accepted 

12.4.3  Access  MOEs 

The  system  shall  provide  a  way  to  access 
old  MOEs 

21  Additional  Out-of-Scope 
Requirements 

12.5.1  Generate  00s 

The  system  shah  allow  generation  of  00s 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.5.2  00  Template 

The  system  shall  provide  a  template  to 
develop  00s 

15  Out  of  Scope 

Spreadsheet  Requirements 

C-17 


Number  &  Name 
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Status 

12.5.3  Save  objectives 

The  system  shall  provide  a  way  to  save  the 
objectives 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.6.1  Generate  Sis 

The  system  shall  allow  generation  of  Sis 

21  Additional  Out-of-Scope 
Requirements 

12.6.2  Save  Sis 

The  system  shall  provide  a  way  to  save  Sis 

21  Additional  Out-of-Scope 
Requirements 

12.6.3  Access  Sis 

The  system  shall  provide  a  way  to  access 
old  Sis 

21  Additional  Out-of-Scope 
Requirements 

12.7.1  Generate  TTs 

The  system  shall  allow  generation  of  TTs 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.7.2  Access  TPFDD 

The  system  shall  allow  access  to  TPFDD 

Accepted 

12.7.3  Access  Adversary 
Information 

The  system  shall  allow  access  to  Adversary 
system  information  (IPB,  DISUM,  INSUM) 

Accepted 

12.7.4  Summary  Friendly 
Forces 

The  system  shall  provide  a  summary  of 
friendly  forces 

Accepted 

12.8.1  Generate  MOPs 

The  system  shall  allow  generation  of  MOPs 

Accepted 

12.8.2  Determine 
collectable  measures 

The  system  shall  aid  in  determining  if 
measures  can  be  collected 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.8.3  Determine  good 
measures 

The  system  shall  aid  in  determining  if 
measures  are  "good" 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.8.4  Determine 
measures  collection  time 

The  system  shall  aid  in  determining  how 
long  it  will  take  to  collect  measures 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.9.1  Multiple  COA 
comparison 

The  system  shall  allow  comparison  of 
multiple  COAs 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.9.2  COA  best  Objectives 

The  system  shall  aid  in  determining  which 
COA  will  best  achieve  objectives 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.9.3  COA  Least  Risk 

The  system  shall  aid  in  determining  which 
COA  produces  less  risk 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.9.4  COA  Ranking 

The  system  shall  allow  the  ranking  of  COA 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.10  COA  Analysis 
methods 

The  system  shall  provide  various  methods 
to  analyze  multiple  COAs 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.11  Prioritize  Op 
Objectives 

The  system  shall  allow  operational 
objectives  to  be  prioritized 

15  Out  of  Scope 

Spreadsheet  Requirements 

Number  &  Name 

Description 

Status 

12.12  Sequence  Op 
Objectives 

The  system  shall  allow  operational 
objectives  to  be  sequenced 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.13  Phase  Op  Objectives 

The  system  shall  allow  Operational 
objectives  to  be  phased 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.14  Determine  WOE  for 
Op  Objectives 

The  system  shall  allow  Operational 
objectives  weight  of  effort  to  be  determined 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.15  Prioritize  TO 

The  system  shall  allow  TO  to  be  prioritized 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.16  Sequence  TO 

The  system  shall  allow  TO  to  be  sequenced 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.17  Phase  TO 

The  system  shall  allow  TO  to  be  phased 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.18  Determine  TO  WOE 

The  system  shall  allow  TO  weight  of  effort 
to  be  determined 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.19  Prioritize  TTs 

The  system  shall  allow  TT  to  be  prioritized 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.20  Sequence  TTs 

The  system  shall  allow  TT  to  be  sequenced 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.21  Phase  TTs 

The  system  shall  allow  TT  to  be  phased 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.22  Determine  TT  WOE 

The  system  shall  allow  TT  weight  of  effort 
to  be  determined 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.23.1  Effect  Priority 
Display 

The  system  shall  display  the  priority  of 
effects 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.23.2  Effect  Sequencing 
Display 

The  system  shall  display  the  sequencing  of 
effects 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.23.3  WOE  Display 

They  system  shall  display  the  Weight  of 
effort 

Accepted 

12.23.4  Effect  Persistence 
Display 

The  system  shall  display  the  Persistence  of 
effect 

18  Rewording  of 
Requirements 

12.23.5  Effect  Location 
Display 

The  system  shall  display  the  location  of 
where  effects  needs  to  take  place 

Accepted 

12.24  Determine  possible 
unintended  effects 

The  system  shall  aid  in  determining 
potential  unintended  effects 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.25  Determine 
Comparison  Criteria 

The  system  shall  aid  in  determining 
comparison  criteria 

15  Out  of  Scope 

Spreadsheet  Requirements 
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Number  &  Name 

Description 

Status 

12.26  COA  rating 

The  system  shall  allow  the  ability  to  rate 
each  COA  against  the  criteria 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.27  COA 
recommendation 

The  system  shall  allow  the  ability  to 
recommend  the  highest  rated  COA 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.28  COA  Modification 

The  system  shall  provide  the  ability  to 
modify  COA 

15  Out  of  Scope 

Spreadsheet  Requirements 

12.29  STTM  Development 

The  system  shall  provide  the  ability  to 
develop  the  STTM 

17  Defective  Originating 
Requirements 

13.1.1  Wargame  COAs 

The  system  shall  provide  the  capability  to 
wargame  COAs 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.1.2  Provide  COA 
Framework 

The  system  shall  provide  a  basic 
framework  for  the  development  of  the 

COAs 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.1.3  COA  Outcome 
Playback 

The  system  shall  "play"  the  outcome  of 

COAs 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.1.4  ID  COA  Weakness 

The  system  shall  identify  weaknesses  in  the 
COA 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.1.5  Display  Friendly 
Losses 

The  system  shall  display  friendly  losses 

Accepted 

13.1.6  Display  Effects 

The  system  shall  display  both  desired 
effects  and  undesired  effects 

Accepted 

13.1.7  Display  Collateral 
Damages 

The  system  shall  display  collateral  damages 
based  on  COAs 

21  Additional  Out-of-Scope 
Requirements 

13.1.8  Display  COA 
options 

The  system  shall  display  options  to  let  user 
choose  the  best  COAs 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.1.9  Reporting 

Mechanism 

The  system  shall  provide  a  reporting 
mechanism 

17  Defective  Originating 
Requirements 

13.1.10  COA  Modifications 

The  system  shall  allow  modifications  to 

COAs 

21  Additional  Out-of-Scope 
Requirements 

13.2  COA  Recalculations 
on  Changes 

The  system  shall  recalculate  the  probability 
of  attaining  commanders  intent  was  well  as 
the  changes  in  the  criteria  values  when 

COAs  are  modified 

21  Additional  Out-of-Scope 
Requirements 

13.3  Creation  of  Sequels 

The  system  shall  allow  the  creation  of 
sequel  plans  that  allow  friendly  forces  to 
capitalize  on  achievement  of  objectives  and 
desired  effects 

15  Out  of  Scope 

Spreadsheet  Requirements 
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Number  &  Name 

Description 

Status 

13.4.1  Aid  in  Undesired 
Effects  Assessment 

The  system  shall  support  and  aid  in  the 
assessment  of  the  likelihood  of  Undesired 
Effects  occurring  given  the  likely  enemy 
reactions 

Accepted 

13.4.2  Adversary  History 

The  system  shall  provide  the  history  of 
adversary 

Accepted 

13.4.3  Adversary  Allies 

The  system  shall  provide  information 
concerning  the  adversary  allies 

Accepted 

13.4.4  Adversary 
Capabilities 

The  system  shall  provide  information 
concerning  adversary  capabilities 

Accepted 

13.4.5  Adversary  Intent 

The  system  shall  provide  information 
concerning  adversaries  intent 

Accepted 

13.4.6  Enemy  Actions  & 
Undesired  Effects 

The  system  shall  aid  in  determining  how 
might  the  enemy  act  to  produce  Undesired 
Effects. 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.4.7  Consequence  of 
Undesired  Effects 

The  system  shall  aid  in  considering  the 
consequences  of  all  known  Undesired 

Effects. 

Accepted 

13.4.8  Undesired  Effects 
mitigation 

The  system  shall  recommend  changes  that 
mitigate  the  risk  of  causing  undesired 
effects 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.4.9  Create  Branch 

Plans 

The  system  shall  create  basic  branch  plans 
that  address  enemy  reactions  and  mitigate 
risks  of  unintended  and  undesired  effects 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.5.1  Visualize  COA 
operations 

The  system  shall  aid  in  the  visualization  on 
how  operations  will  unfold  based  on  the 
selected  COA 

Accepted 

13.5.2  COA  Path 

The  system  shall  aid  in  determining  if  COA 
path  is  correct 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.5.3  COA  Path  Mods 

The  system  shall  aid  in  determining  if  COA 
needs  to  modified 

19  Update  Requirements 
to  Reflect  Analysis 

13.5.4  COA  Mission 

Analysis 

The  system  shall  aid  in  determining  if  COA 
will  achieve  objectives. 

19  Update  Requirements 
to  Reflect  Analysis 

13.5.5  COA  Path  Effects 

The  system  shall  aid  in  determining 
undesired/desired  effects  along  pathway 

20  Additional  Defective 
Requirements 

13.6.1  COA  Commanders 
Intent 

The  system  shall  aid  in  the  comparison  of 
COA  options  to  commanders  intent  to 
ensure  intent  is  being  met 

15  Out  of  Scope 

Spreadsheet  Requirements 
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Number  &  Name 

Description 

Status 

13.6.2  COA  Measures 

The  system  shall  aid  in  determining  if  COAs 
are  measurable  for  assessment 

20  Additional  Defective 
Requirements 

13.7.1  Mission  Archives 
Access 

The  system  shall  provide  access  to  archival 
data  concerning  past  mission 

Accepted 

13.7.2  Mission  Archives 
Search 

The  system  shall  provide  a  way  to  search 
for  archives 

Accepted 

13.7.3  Data  Archive 

Creation 

The  system  shall  provide  a  way  to  archive 
data 

Accepted 

13.7.4  Archive  Search  Tips 

The  system  shall  provide  tips  on  how  to 
search  for  archives 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.8.1  Building  Timeline 

The  system  shall  aid  in  building  a  timeline 
to  identify  when  certain  objectives  are 
projected  to  occur 

20  Additional  Defective 
Requirements 

13.8.2  Event  Occurrence 
Plotting 

The  system  shall  plot  and  aid  in 
determining  when  events  should  occur 

Accepted 

13.8.3  Event  Complete 
Plotting 

The  system  shall  plot  and  aid  in 
determining  when  events  should  be 
complete 

Accepted 

13.8.4  Event  Sequencing 
and  Plotting 

The  system  shall  plot  and  aid  in 
determining  what  events  precede  others 

Accepted 

13.9  COA  Wargaming 
Analysis 

The  system  shall  aid  in  identifying 
advantages  and  disadvantages  of  each  COA 
based  on  wargaming 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.10.1  COA  Refinement 

The  system  shall  allow  refinement  of  each 
COA 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.10.2  Develop  likely 
enemy  reactions 

The  system  shall  aid  in  developing  enemy's 
most  likely  and  most  dangerous  reactions 
based  on  recommendations,  sequel  and 
branch  plans 

Accepted 

13.10.3  FrOB  Validation 

The  system  shall  aid  in  validating  FrOB 
based  on  COA  wargaming 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.10.4  Force  Mix 
Recommendation 

The  system  shall  recommend  a  different 
force  mix 

15  Out  of  Scope 

Spreadsheet  Requirements 

13.10.5  Resource  Ranking 

The  system  shall  aid  in  and  allow  the 
ranking  of  resources  according  to  expected 
contributions  of  friendly  and  adversary 
forces  to  achieving  desired  effects 

15  Out  of  Scope 

Spreadsheet  Requirements 
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Number  &  Name 

Description 

Status 

13.11  Projected  Task 
Timeline 

The  system  shall  display  when  certain  tasks 
are  projected  to  occur 

19  Update  Requirements 
to  Reflect  Analysis 

13.12  Projected  Effect 
Timeline 

The  system  shall  display  when  effects  are 
projected  to  occur 

19  Update  Requirements 
to  Reflect  Analysis 

13.13  Analysis  Linkage 
Development 

The  system  shall  aid  in  developing 
linkage(s)  to  support  a  timely  and  accurate 
analysis 

15  Out  of  Scope 

Spreadsheet  Requirements 

14.1.1  Objective  Data 
access 

The  system  shall  have  access  to  Objectives 

Accepted 

14.1.2  Task  Data  access 

The  system  shall  have  access  to  Tasks 

Accepted 

14.1.4  Task  &  Objective 
access 

The  system  shall  allow  have  access  to  all 
measures  associated  with  tasks  and 
objectives 

Accepted 

14.1.5  Measure  Changes 

The  system  shall  allow  changes  to  be  made 
to  measures 

Accepted 

14.2  View  AOD  data 

The  system  shall  provide  the  capability  to 
view  JFACC  objectives,  tasks,  and  their 
associated  measures,  and  the  AOD 

18  Rewording  of 
Requirements 
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1  Scope  &  Purpose 

'iTiis  document  defines  the  underlying  principles  and  derived  guidelines  used  to  generate 
the  OEAV'l’  concept  displays  that  include  cartographic  display  elenienLs  (i.e.^  maps) 
either  as  a  priiucuy^  display  or  a  eontextiml  background. 

2  References 

Snyder^  John  P.  1987.  Kdap  projections:  a  working  manual  USGS  Professional  Paper 
1395.  Washington,  DC;  United  Slates  Government  Printing  OlHee 
TuftCj  Edward  R.  1997.  Visual  Explanations:  Images  and  Quantities,  Evidence  and 
Narrative.  Cheshire,  CT:  Graphics  Press 

Tulle,  Edward  R.  2001.  The  Visual  Display  of  Quantitative  Informaiion.  T'^Ed. 

Cheshire,  C'P;  Ciraphics  Press 

HolTman,  Gemot,  2005.  CIE  Lab  Color  Space 

http ;  // WWW.  Iho-  emde  n.  de/  -holl  mann/c  ie  lab03022 0 03 .  pdf 

lVIlL-STn-2525H  w/CllANGE  1,  Common  Warfighting  Symbology,  1  July  2005 
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3.0  Functional  use  of  maps  within  OEAVT 

Maps  in  (IKA  VT  are  used  to: 

Place  symbolic  information  overlays  in  geographic  context 
Display  spatial  relationships  between  symbolic  elements  Maps  in  Oil  AVI’  are 
specifically  not  used  to: 

Pcrfbnn  quant  itat  i  ve  c  va  luation  of  terrain  (e .  g. ,  cont  our  1  incs  ) 

[display  precise,  quantifiable  spatial  relatioiLsbips  appropriate  for  targeting, 
specific  target  I  ID  A,  or  detailed  planning  of  operational  movements. 

To  meet  these  functional  objectives  in  OEAVT,  several  design  principles  for  map 
displays  are  defined  in  section  3.  1’ablc  3.0  summarizes  these  design  recommendations. 


Table  3.0  Summary  of  Map  Display  Design  Recommendations  for  OEAVT 


[l-  . 

Fimclion 

Recommendalion 

Comment 

3. 1 

Display  sie^i  &  colors 

Single  display,  1024xT<5S,  millions 
of  colors 

Should  support 
multiple  displays  and 
larger  monitors  if 
available. 

3.2,1 

Map  datum 

WGS-84 

3.2.1 

Projcelion 

Always  use  projected  map  displays 

3.11 

Projection:  large^area,  mid- 
latitude 

Lambert  Conformal  Conic 

Typically  30 ' 80 
degrees 

3.2.1 

Projection:  laigc-arca,  cquatcfrial 

Mercator 

Typically  i30  of 
equator 

3.11 

Projection:  large-area,  polar 

Universal  Polar  Stereogmphic 

Typically  above  80 
degrees 

3.11 

Projection:  small-area  (except 
polar) 

Universal  Transverse  Mercator 

3.2,1 

Projection:  special  use 

Orthographic 

Used  for  planetary 
hemisphere  views 

3.12 

Depiction  of  terrain 

Shaded  relief 

Preserve  colorspace 
and  line  graphics  for 
data  other  than  terrain. 

3.12 

Terrain  vertical  cxaggcmlion 

Avoid.  Limit  to  3x  if  needed,  and 
iabe]_the  exa^erated  graphic. 

3.2.2 

T  erra  in  rel  te  f  light  ing 

Always  within  45  degrees  of 
directly  '"above"*  from  the  user's 
point  of  view. 

For  north-up  maps,  the 
light  source  for  shaded 
relief  w  ill  be  located 
between  NW  and  HE. 

3.2.3 

Map  background  color  pallet 

Use  a  color  pallet  >50  “L"  in  the 

CIE  L^a^b  colorspace  for  map 
background  colors  such  as  lEind 
cover  and  water  combined  with 
shaded  relief 

Goal  IS  vbuul  contrast 
between  backgKT'Und 
imip  and  cjverlays  in 
both  displays  and 
printed  copy 

3.14 

1 

Use  of  photographic  imagery 

Use  only  for  high-detail,  small  area 
maps. 

3.15 

Imagery  metadata 

OEAVT  displays  should  maintain 
the  identity  metadata  of  display 
imagery 

3.15  1 

PhotOfu^phic  imagery  rendcritig 

Use  a  reduced  pallet  ofL  <  50. 
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Function 

Recommendation 

Comment 

3.16 

Graphic  layers 

Display  order  and  appearance  of 
graphic  layers  should  be  user 
adjustable. 

3.2.6 

Graphic  layers 

Order  of  layers  in  a  control  list 
should  reflect  the  display  order  of 
precedence  (c.g.,  upper  layers  may 
obscure  lower  layers) 

3.2.6 

Terrain  laj'er 

Should  always  be  the  lowest  la>'er 

3.2.6 

Areal  tint  layer 

Next  layer  above  terrain,.  Depicted 
as  reduced  pallet  transparent  lints 
over  terrain  shaded  relief  layer. 
Content  is  user  selectable 

3.2.6 

Photographic  layer 

Next  layer  above  areal  tint  layer 

Use  of  photographs 
obscures  areal  tints 
and  shaded  relief. 

3.16 

Map  overlay  selectable 
parameters 

Stacking  order.  Presence,  De- 
Clutter,  Areal  layer  content. 

Overlay  color/symbol,  gridlines. 

3.16 

1 

Map  scale 

Scale  should  always  be  displayed 
using  units  of  mete  rs  or  k  i  lom  eters 

3.16  ^ 

North  Arrow 

No  arrow  should  be  ased. 

All  maps  displayed 
Norlh-up 

3.17 

Display  of  lime 

Continuous  animation  of  symbols 
or  time-slice  snapshots. 

IT.7  i 
3.17  i 

1 

User  animation  controls 

Symbols  in  a  time-referenced 
display 

Playback  speed  and  direction 
Thumbnails,  time  scale,  indication 
of  time  depicted  in  jTi am  display 

Tuftc  example 
illustration. 

3,1  Map  display  constraints 

Map  displays  in  OEAVT  are  generated  using  CO'l’S  computers  lypieally  ninning 
Windows  2000  or  Windows  XP  operating  systems.  The  lypieal  OEAVT  user  will  have 
tw^o  desktop  screens  available,  but  may  be  limited  to  a  laptop  computer  wHth  a  single 
display. 

The  map  display  should  be  designed  to  require  no  more  than  1024  x  76S  pixel 
display,  with  24  bit  RGB  color  (e.g,,  millions  of  colors). 

Ideally,  the  map  display  will  retain  full  functionality  if  displayed  as  a  sixteen- 
bit  color-mapped  display  (e  g.,  thousands  of  colors), 
file  map  should  support  presentation  as  a  window^  wdthin  the  display 
constraints  identified  above. 

The  map  display  should  support  display  on  a  separate  monitor,  using  all  or 
part  of  the  resolution  of  that  monitor. 

fhe  map  display  should  support  display  on  larger  format  monitors,  e.g.  12K()  x 
1 024  and  larger  without  adverse  effects. 
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3-2  Visual  Design  Guideline  for  Maps 
3.2.1  Map  Projection  and  Datum 

A  map  inherently  distorts  the  spatial  relationship  of  objects  on  a  spherical  surface.  The 
degree  of  the  distortion  depends  upon  the  scale  of  the  map,  the  latitude  of  the  map 
projection  center,  and  the  selected  map  projection.  Maps  should  never  be  displayed  in  an 
un-projected  format.  See  figure  3- 1  for  an  example  of  Europe  and  Western  Asia 
displayed  both  un-projected,  and  projected  using  a  Lambert  Conformal  Conic  projection. 
All  OEAVT  map  displays  should  be  referenced  to  the  WGS-84  datum  horizontal  datum 
and  mean  sea  level  for  the  vertical  datum. 


Figure  3-1:  Example  of  an  un-projected  versus  a  projected  map. 

The  following  map  projections  are  recommended  for  use  in  OEAVT,  as  the  projections 
are  already  in  use  in  operational  graphics; 

Lambert  Conformal  Conic:  This  projection  is  recommended  for  small-scale  (large  area) 
mid-latitude  maps,  as  well  as  large-scale  (small  area)  maps.  The  standard  parallels 
should  be  selected  to  minimize  distortion  across  the  selected  display.  This  is  the  standard 
projection  for  most  mid-latitude  small  scale  graphics  such  as  Joint  Operational  Graphics 
(JOGs)  created  by  the  National  Geo- Spatial  Intelligence  Agency  (NGA). 

Mercator:  This  projection  is  recommended  for  small-scale  maps  in  equatorial  regions. 
Outside  of  equatorial  regions,  Mercator  is  a  special-use  projection  due  to  large 
distortions. 

Universal  Transverse  Mercator  (XJTM):  This  projection  is  recommended  for  large-scale 
(small  area)  maps  when  correlation  with  overlays  inherently  expressed  in  UTM  grid 
coordinates  is  required  (e.g.,  phase  lines,  etc.). 

Universal  Polar  Stereographic  Projection:  This  projection  is  recommended  for  both 
large-scale  and  small-scale  maps  of  the  polar  regions  north  of  84°  latitude  or  south  of  - 
80°  latitude. 

Orthographic:  This  projection  may  serve  as  a  special-use  map  display  of  planetary 
hemispheres.  Especially  if  animated  for  user  selection  of  the  projection  center,  the 
orthographic  projection  is  useful  for  overview  and  selection  of  more  specific  areas  of 
interest  to  be  displayed  by  one  of  the  other  recommended  projections.  Figures  3-2 
depicts  an  example  of  an  un-projected  rendering  of  topographic  height  and  bathometric 
depth,  while  figure  3-3  depicts  an  example  of  the  Orthographic  projection  (single 
hemisphere)  from  the  same  dataset. 


4 


D-IO 


5795-OEAVT-Task2-Map  Displays-vl  .0 


Additional  information  on  these  projections  including  a  complete  mathematical  definition 
may  be  found  in  Snyder,  1987. 


Figure  3-2:  An  un-projccted  rendering  of  the  K'l'OPO-S  dataset  (topographic  height 
and  bathymetric  depth  on  a  5  arc  minute  spacing).  Note  the  extreme  distortion  in 
the  polar  regions.  Only  the  equator  is  true-scale. 


Figure  3-3:  An  orthographic  projection  rendering  of  the  same  ETOPO-5  data  set 
with  the  projection  centered  on  North  America. 

2.2.2  Display  of  terrain 

Militar>’  planners  must  consider  the  impact  of  terrain  on  operations.  Although  OEAVT 
displays  will  not  be  used  for  detail  planning,  major  terrain  features  and  charaeteristies 
must  be  depicted  in  order  to  simplify  correlation  with  other  operational  map  graphics 
used  in  the  planning  and  evaluation  process.  Several  methods  of  terrain  depiction  are 
identified  below. 
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Shaded  Relief:  This  method  is  recommended  for  OEAVT  displays,  as  it  allows 
qualitative  assessment  of  gross  terrain  features  at  a  glance.  A  shaded  relief  map  in  gray 
scale  allows  for  the  shading  to  be  merged  with  overlay  color  tints  containing  non-terrain 
information  that  may  be  needed  to  enhance  assessment  such  as  land  cover,  precipitation, 
population  density  or  ethnicity  among  others.  Some  cautions  are  in  order  for  the 
implementation  of  shaded  relief  terrain  maps.  The  light  source  must  appear  to  be 
overhead  to  the  observer  less  terrain  features  appear  inverted.  For  a  North-up  map,  this 
means  a  light  source  positioned  between  NW  and  NE.  Also,  vertical  scale  exaggeration 
may  be  used,  with  caution,  to  enhance  terrain  relief  It  is  recommended  that  vertical 
exaggeration  be  limited  to  no  greater  than  3x,  and  applied  to  all  map  displays  on  a  user- 
selected  on-off  basis.  Selectively  scaling  one  region  and  not  another  will  lead  to 
confusion  regarding  actual  terrain  Examples  appear  in  figure  3-4  below. 


Figure  3-4:  Shaded  Relief.  The  figures  above  depict  hills  in  generally  flat  Ohio 


terrain  (roughly  300’  hilltop  to  valley  floor  maximum  difference).  The  figure  at  left 
depicts  lighting  from  the  North  West  (above,  left  relative  to  the  reader),  and  no 
vertical  exaggeration  of  the  terrain.  The  center  figure  depicts  lighting  from  the 
North  West,  and  a  3x  vertical  exaggeration.  Note  the  improved  perception  of  hills  to 
valleys  in  the  middle  figure.  The  right  figure  depicts  lighting  from  the  South  East 
(from  below -right  relative  to  the  reader),  and  3x  vertical  exaggeration.  Note  the 
reversal  of  the  perceived  high/low  relationship  with  lighting  from  “below”.  The 
central  valley  apparent  (correctly)  in  the  center  figure  is  perceived  as  a  sinuous 
plateau  in  the  right  figure. 

Contour  Lines:  This  method  allows  precise,  quantitative  evaluation  of  terrain.  Contour 
lines  are  frequently  combined  with  shaded  relief  on  tactical  operations  maps  in  an  effort 
to  combine  at-a-glance  qualitative  understanding  of  the  terrain  features  with  the  ability  to 
perform  quantitative  analysis.  Examples  of  contour  lines  with  and  without  relief  shading 
are  depicted  in  figure  3-5.  Contour  lines  in  either  form  are  not  recommended  for  OEAVT 
displays  as  the  precision  is  unnecessary,  and  the  lines  add  distracting  visual  clutter  to 
necessary  overlays. 


6 


D-12 


5795-OEAVT-Task2-Map  Displays-vl  .0 


Figure  3-5:  Examples  of  terrain  depicted  by  contour  lines  on  a  USGS  7.5’  series 
topographic  map.  The  left  figure  depicts  contour  lines  only,  the  right  figure  adds 
shaded  relief  to  aid  qualitative  understanding.  The  terrain  depicted  is  a  detailed 
view  of  the  hill/valley  terrain  previously  depicted  in  shaded  relief. 

Gray  scale  coded  elevation:  This  method  is  difficult  for  most  users  to  interpret,  and  its 
use  in  not  recommended  for  OEAVT.  The  figure  3-6  depicts  a  gray-scale  encoding  of 
elevation,  and  a  corresponding  contour  map. 


Figure  3-6:  Gray  scale  encoding  of  elevation  is  depicted  to  the  left  of  a  contour  map 
of  the  same  region.  Note  that  the  contour  map  depicts  the  central  portion  of  the 
gray  scale,  not  the  entire  area.  The  terrain  depicted  is  the  same  region  of  hill/valley 
terrain  previously  depicted  in  shaded  relief. 

Color- coded  elevation:  This  method  is  especially  useful  when  combined  with  shaded 
relief  The  designer  is  cautioned  that  local  optimization  of  color  pallet  to  best  reveal 
subtle  terrain  changes  can  cause  confusion  when  the  same  pallet  is  re-used  for  different 
terrain.  If  color-coded  terrain  is  used,  then  an  absolute  terrain  elevation  to  color  pallet 
capable  of  depicting  world- wide  elevations  should  be  used,  with  shaded  relief  added  to 
enhance  the  perception  of  terrain.  An  unsaturated  (e.g.  pastel)  pallet  is  recommended  for 
OEAVT  maps,  following  the  design  principle  of  minimum  differences  described  in  Tufte, 
2002.  This  unsaturated  reference  background  improves  the  readability  of  more  saturated 
overlays  of  effect  evaluation  symbology. 
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Color-encoded  elevation  Color  coding  +  shaded  relief  Absolute  scale  +  shaded  relief 

Figure  3-7:  Color  coded  depictions  of  terrain  elevation. 

The  three  panes  of  figure  3-7  depict  terrain  elevation  encoded  by  color.  The  left  figure 
depicts  a  color  encoding  scale  optimized  for  a  single  region  (uses  the  entire  range  of  the 
pallet  for  the  elevations  present).  The  center  figure  depicts  the  addition  of  shaded-relief 
to  the  optimized  encoding  scale.  The  right  figure  depicts  an  absolute  color  encoding 
scale  (optimized  for  world-wide  elevations),  combined  with  shaded  relief 


2.2.3  Display  of  land/water  features:  limited  color  pallet 


Water  features  should  be  depicted  in  a  blue  pallet  to  provide  color-contrast  to  land 
features,  which  are  traditionally  depicted  in  a  pallet  of  browns  or  greens.  The  blue 
saturation  selected  for  water  features  should  be  scaled  mid-range  to  the  value  of  the  land 
pallet.  If  bathometric  depths  or  other  variations  must  be  expressed  within  the  water  layer, 
it  is  recommended  that  an  unsaturated  (e.g.,  pastel)  color  pallet  be  used  for  both  land  and 
water.  Ideally,  the  darkest  color  in  both  land  and  water  pallets  will  not  drop  below  a 
value  for  “L”  of  fifty  as  measured  in  the  CIE  L*a*b  color  space  (where  L  expresses 
‘‘psychometric  brightness”  in  a  scale  from  zero  (maximum  darkness)  to  100  (maximum 
lightness).  Eigure  3-8  depicts  a  such  a  pallet  depicting  land/water  contrasts. 
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Figure  3-8:  Land  topographic  heights  and  water  bathometric  depths  portrayed  in 
an  example  limited  range  pallet.  Such  a  pallet  reserves  saturated  colors  for 
overlays,  as  depicted  by  the  overlay  arrows.  Note  they  remain  distinct  despite 
crossing  a  wide  range  of  pallet  values.  Picking  a  text  similar  in  perceived  brightness 
but  distinct  in  color  distinct  from  dtlier  the  land  or  water  pallets  preserves  the 
readability  of  text  without  ovendiehning  visual  dutter. 

2.2.4  Use  of  overhead  imagery  to  depict  iand/water 

Projected  overhead  imagery  may  form  the  map  base  display.  This  is  especially  useful  for 
large-scale  (small  area)  depictions  where  cultural  features  such  as  buildings,  earthworks 
and  roads  become  more  descriptive  of  the  operational  ‘‘terrain”  than  hare-earth  terrain 
depictions  generated  by  databases.  If  used  as  a  base  map,  the  saturation  of  imagery 
should  be  reduced  to  an  L  value  of  less  than  fifty.  In  general,  panchromatic  imagery  is 
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preferred  over  iiuilti -spectral  imager\'  for  use  as  a  background  map  due  to  reduced  visual 
clutter. 

2,2.5  Scale  issues  in  map  graphics  and  Imagery 

For  small  scales,  reference  maps  consisting  of  symbol ie  abstractions  of  physical  and 
cultural  features  are  recommended  over  photographic  images  such  as  Clli.  lliis  is  due  to 
the  presence  of  visual  clutter  in  small  scale  photographs  that  impede  understanding  of 
relevant  terrain  features,  as  depicted  in  figure  3-9  below  by  a  1:24000  map  rendering  (the 
largest  scale  map  likely  to  be  encountered  in  nnlitary'  operations  or  derived  from  NG.4 
map  databases),  and  the  eorresponding  area  in  a  one  meter  aerial  pholograph. 

At  even  larger  scales,  depending  upon  terrain  feature  density,  symbolic  maps  become  less 
useful  as  more  and  more  of  the  displayed  area  is  occupied  by  featureless  while  space 
isolated  symbols  without  adequate  spatial  reference.  At  vciy'^  large  scales^  typically  less 
than  1:50000  for  suburban  or  urban  lerraiii  bul  perhaps  up  to  1:100,000  for  more 
featureless  desert  terrain,  photographic  depictions  become  preferable  as  the  increased 
eonleiit  provides  more  useful  refereuee  clues.  In  the  illustration  below,  a  small  portion  of 
a  built-up  area  is  identified  using  a  red  rectangle  bos.  lliis  area  is  depicted  enlarged  as  a 
one-meler  resolution  photographic  image  in  figure  3-10. 

Note  that  for  this  small  area,  the  pbotograpbic  image  provides  a  more  detailed 
understanding  of  the  selected  area,  without  so  much  detail  as  to  be  overw  lielming  as  in 
the  smaller  scale  examples.  11’ photographic  images  are  selected  to  depict  terrain  features, 
a  limited  pallet  will  again  enhance  the  readability'  of  overlays  that  represent  the  core 
content  of  OEAVT  displays. 

If  imagery  is  presented,  there  is  the  risk  that  assessors  w'ill  attempt  to  exploit  the  content 
of  the  imagery  (e.g,  is  that  runway  still  serviceable,  is  that  SAM  site  operational,  etc.), 
rather  than  use  the  image  simply  as  a  spatial  reference,  lliis  is  inappropriate  and 
potentially  dangerous.  Oh^AV'f  displays  of  imagery,  especially  intelligence  imagery, 
should  alw'ays  contain  metadata  that  maintains  the  identity  of  the  source  image. 

'llic  metadata  allows  retrieval  of  the  source  image  so  that  it  can  be  exploited  using 
appropriate  tools  and  metliods  if  necessary'.  Excellent  tools  exist  for  the  detailed 
exploitation  of  imagery';  an  assessment  display  should  not  lure  the  user  into  using  a 
background  photographic  image  for  exploitation  rather  than  reference,  llic  use  of  a 
reduced  contrast  pallet  as  depicted  in  the  figure  below  both  aids  readability  of  overlays 
and  discdurages  inappropriate  use  (attempted  e.Kpltnlatitni)  of  the  photographic  image. 
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Figure  3-9:  Comparison  of  1:24000  map  feature  density  to  1m  resoLution  aerial 
phutu^raph.  Nule  that  Ihe  reduced  visual  information  on  the  map  enables  more 
rapid  comprehension  of  key  terrain  and  cultural  features. 
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Figure  3-10:  Reduced  pallet  rendering  of  photographic  images  improves  legibility 
of  overlay  s/symbols,  using  the  image  for  spatial  reference,  not  exploitation. 
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2.2.6  Use  of  selectable  layers 

OEAVT  maps  should  be  displayed  in  layers,  witli  ''upper”  layers  capable  oF  obscuring 
symbols  on  lower  layers.  Layers  displayed  should  always  be  defined  in  a  map  legend, 
'llie  airangement  of  layers  should  be  user  selectable  with  ihe  exception  of  lliree 
pennaneiil  *'area”  feature  layers  making  up  the  base  map  as  depicted  below  (base  layers 
are  underlined). : 

»  Upper  user-select  able /user-stackable  symbol  layers 

» “  “ 

»  Photoirranhic  background  layer  (obscures  lower  layere  if  selected). 

The  content  for  this  layer  may  be  derived  from  standard  NCjA  products 
such  as  Controlled  linage  Base  (CIB),  or  from  operational  imagery  . 

»  Areal  'lint”  layer  to  sliadcd  relief  (non-ohsenring).  Note  that  the 
content  and  presence  of  this  layer  is  iLser  selectable. 

» 1  errain  base-layer,  typically  depicted  as  shaded  relief,  and  derived 
from  standard  NCiA  sources  (e.g,,  D'fI  J>) 

Layered  map  overlays  should  be  user  selectable  for  the  following  parameters: 

Stacking  order:  Where  symbols  on  user- defined  layers  may  obscure  symbols 
on  lower  layers  and  the  base  layers,  llic  order  of  layers  in  the  map  legend 
should  correspond  to  the  stacking  order. 

Presence:  layers  may  be  turned  on/or  off.  If  turned  off Ahey  should  be 
removed  (or  grayed)  in  the  map  legend. 

l>e-clutter:  A  de -clutter  capability  should  he  provided  to  allows  all  other  layers 
other  than  the  selected  layer  and  the  base  layers)  to  be  temporarily  removed  or 
grayed  (visually  minimized).  Tlie  de-cl uttered  layer  should  not  bo  obscured 
by  any  other  map  layer. 

Areal  tint  layer  content:  Tlie  tint  applied  to  the  sliaded  relief  terrain  may 
depict  a  w^ide  range  of  information:  natural  and  cultural  land  cover 
classification,  population  density,  ethnic  and  religious  populations,  sensor 
/threat  coverage,  weather,  etc.  The  tints  applied  should  provide  a  default 
pallet,  with  further  user- adjustment  possible. 

tK'crlay  col  or/symbol:  A  pallet  of  over- lay  colors/symbols  compliant  with 
MIL- STD  2525B  should  be  provided  for  each  point  and  line  feature  overlay. 
Each  symbol  type  should  also  have  a  selectable  ^'subdued'’  color,  derived  from 
its  standard  color.  Tlic  design  intent  of  the  subdued  color  is  to  continue  to 
have  the  context  of  the  seiccled  symbol  present  on  the  display,  but  at  a 
reduced  clutter  level. 

CJridlincs:  Ciridlines  in  either  latitude/1  ongitiide  or  IJ  fM  Northing/Ea.sting 
should  be  selectable  on/olfand  available  in  both  contrasting  and  subdued 
colors.  All  gridlines  should  include  numeric  values.  Gridline  spacing  should 
allow  both  an  aulo-spacing  (dividing  the  map  into  a  4x4  grid),  and  user- 
defined  grid  line  spacing  intci  vals  for  both  latitudc/longitude  or  UTM. 
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Purpose: 


This  trade  study  examines  the  functional  capability  of  severai  candidate  mapping 
applications  against  the  notionai  functionai  requirements  of  the  OEAVT  Geospatiai 
Effects  Visualization  (GEV).  Both  open-source  tools  familiar  to  the  OEAVT  team 
and  commercial  mapping  tooi  kits  are  examined. 

The  trade  study  has  been  tailored,  at  customer  request,  to  allow  a  similar 
comparison  against  the  COA  Sketoh  mapping  requirements  (COA  Sketch  is  a 
Systems  Research  Associates  deveiopment  a  iso  conducted  by  AFRL).  The  gcai  of 
this  taiioring  is  to  examine  the  suitability  of  a  singie  mapping  application  to  meet  both 
OEAVT  and  COA  Sketch  needs. 

Scope: 

The  trade  study  recommendation  is  iimited  to  the  initiai,  prototype  deveiopment  and 
deployment  of  the  GEV. 
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Mapping  Tools  -  Functional  &  Requirement  Comparison 

Table  1:  Functions  mapped  as  M.  Mandatory,  HD  Highly  Desired,  0  Desired,  and 

NS,  Not  Significant  to  program. 


Feature 

OEAVT  GEV 

COA  Sketch 

GUI  /  Interface  Capabilities 

1,  Configurable  GUI 

M 

2,  Able  to  embed  into  separate 
applications 

M 

3.  3D  interface 

NS 

4  3D  terrain  rendering 

NS 

Software  Libraries 

5,  C++  SDK 

NS 

6.  Java  SDK 

HD 

Drawing  Capabilities 

7,  Raster 

M 

S.  Polygons 

M 

9,  Polylines 

M 

10,  Icons 

M 

Layers 

11.  Visibility 

M 

12  Draw  order 

M 

13.  GUI  Controls 

HD 

File  Types 

14.  ESR I  (GIS  shape  data) 

HD 

15.  DIED 

M 

16.  CADRGandCIB 

M 

17.  Basic  Rastar  (JPG, BMP) 

HD 

18.  Comma  Separated  (CSV) 

D 

1 9  KML  (vector  and  raster  in  XML 

format) 

D 

Projections 

20  Orthographic 

D 

21.  Equal  Arc  (LLXY/CADRG) 

NS 

22.  Mercator 

NS 

23.  Gnomic 

NS 

24  Polar  Stereographic 

HD 

25.  Lambert  Conformal  Cone 

HD 

26.  UTM/MGRS 

HD 

General  Performance  and 

Dependencies 

27.  Speed 

HD 

28.  Reliability 

HD 

29  Validated  Algorithms 

HD 

30.  Internet  Independent 

HD 
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Elaboration  on  Mandatory  and  Highly  Desired  features  for  GEV: 

Mandatory: 

Configurable  GUI:  GEV  concepts  are  layer-based,  with  a  customized  GUI 
providing  ‘'at  a  glance’'  understanding/access  to  the  operational  hierarchy  as  well  as 
control  over  the  displayed  graphics. 

Able  to  embed  into  separate  applications:  The  mapping  engine  is  exactly  that  -  a 
display  tool.  The  GEV  has  underlying  capabilities  for  which  the  map  is  only  a 
visualization. 

Drawing  Capabilities  -  Raster:  Essential  for  rendering  base  map  data,  as  it  is 
commonly  provided  as  CADRG,  can  be  rendered  DTED,  or  may  be  overhead 
imagery. 

Drawing  Capabilities  -  Polygons:  Essential  for  efficiently  displaying  and  editing 
operating  and  assessment  regions  and  other  areal  features. 

Drawing  Capabilities  -  Polylines:  Essential  for  depicting  boundaries,  phase  lines, 
road  and  other  networks,  etc. 

Drawing  Capabilities  -  Icons:  Essential  for  point-feature  depiction.  GEV  will 
employ  predefined  assessment  symbols  places  as  icons  on  the  base  map. 

Layer  Control  -  Visibility:  Essential  for  de-clutter  of  complex  operations  Should 
be  directly  controllable  via  the  GUI. 

Layer  Control  -  Draw  Order:  Fundamental  to  the  visualization  concept  of  the 
GEV.  In  appropriate  draw  order  can  result  in  concealing  assessment  information 
leading  to  erroneous  interpretation. 

DTED,  CADRG,  CIB:  DTED.  CADRG  and  CIB  are  all  raster  formats  and  are  the 
standard  base-map  graphics  in  use  in  the  DoD.  These  are  essential  for  base-map 
layers 

Highly  Desired: 

Java  SDK:  Allows  multi-platform  development  for  both  legacy  UNIX  or  Windows 
deployment  within  the  AOC. 

Read  ESRI  Shape  files:  A  GIS  standard  for  the  interchange  of  linear  and  areal 
features  such  as  roads,  political  boundaries,  etc. 

Read  basic  raster  data:  JPEG  and  other  commercial  raster  formats  are  desired  for 
ingesting  pre-processed  raster  layers  edited  by  external  tools. 
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Polar  Stereo  graphic  Projection:  Standard  projection  used  by  the  USAF  for  high- 
latitude  map  displays. 

Lambert  Conformal  Conic  Projection:  Standard  projection  used  by  the  USAF 
from  mid-latitude  to  equatorial  maps,  including  GNC,  JNC,  JOG  and  TPCs.  Already 
familiar  to  USAF  rated  personnel. 

UTM/MGRS:  The  UTM  projection,  especially  as  implemented  in  the  Military  Grid 
Reference  System,  would  simplify  visualizing  joint  operations  that  were  US  Army¬ 
centric.  USA  fire-support  and  coordination  lines  are  typically  given  solely  in  MGRS. 

Speed:  Especially  critical  for  user  acceptance  as  the  data  becomes  more  complex 
in  large  operations. 

Reliability:  Essential  to  user  acceptance.  Nobody  likes  using  tools  that  crash  and 
lose  work. 

Validated  Algorithms:  For  precision  mapping  work,  this  is  essential,  and  for  many 
GIS  implementation  would  be  mandatory.  For  the  GEV,  since  terrain  analysis  is  not 
an  expected  user  function  and  oniy  general  interpretation  of  the  geospatial 
environment  is  needed,  this  is  downgraded  to  highly  desireable. 

Internet  Independent:  A  resiliency  issue  based  on  the  threat  of  network  attack. 
While  collaboration  is  also  highly  desired,  the  ability  to  continue  using  the  application 
from  local  cached  data  is  viewed  as  highly  desirable. 


Table  2:  Feature  Comparison  by  GIS  Tool 


KEY 

OM 

OpenMap 

AG 

ArcGIS  (ESRI  through  C/JMTK) 

OS 

OSSim 

GEp 

GoogleEarth  pro 

GEb 

GoogleEarth  basic 

GM 

Google  Maps 

JV 

JView 

uD 

uDig 
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Table  2:  Feature  Comparison  by  GIS  Tool 


Feature 

OM 

AG 

OS 

GEp 

GEb 

GM 

JV 

uD 

GUI !  Interface  Capabilities 

1  Configurable  GUI 

X 

X 

X 

2.  Able  to  embed  into  separate 
applications 

X 

X 

3  3D  interface 

X 

X 

4  3D  terrain  rendering 

X 

X 

X 

Software  Libraries 

5.  C++  SDK 
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Notes: 


Partial  -  there  is  only  control  of  raster  imagery;  there  is  no  control  over  poly,  line  or 
icon  layers.  Aiso,  raster  imagery  always  shows  up  above  polys  and  lines,  and 
always  shows  up  below  icons  as  well  as  other  Google  Earth  native  data  like  roads. 
Draw  order  is  not  dependent  on  ordering  in  the  folders  or  the  KML. 

^  Unable  to  overlay  raster  imagery  in  this  projection. 

^  These  capabilities  are  easily  codable,  but  do  not  come  “out-of-the-box." 
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OpenMap 

Pros:  OpenMap  is  an  open  source  library  that  can  be  used  either  as  an  API  to 
incorporate  into  the  user’s  application,  or  as  a  fully  functional  standalone  application. 
Almost  every  aspect  of  it  is  configurable— from  the  GUI  layout  and  functionality  to 
the  data  handling  of  each  layer 

Cons:  OpenMap  is  limited  to  2D  mapping  capabilities,  virtually  all  information  must 
be  provided  by  the  user,  and  it  has  observed  issues  with  speed  and  reliability, 
specifically  when  dealing  with  large  amounts  of  raster  data  such  as  DTED  or 
CADRG 

Licensing:  Open  Source;  see  http://openmap  bbn  com/license  html 

Notes: 

■  Oddly  enough,  even  though  OpenMap  has  ready  made  layers  for  mapping- 
specific  data  such  as  DTED  and  RPF,  there  aren’t  pre-made  layers  for 
generic  raster  and  shape  layers.  These  would  have  to  be  coded  from  basic 
layers. 

•  The  OpenMap  Architecture  is  described  here: 

http://openmap.bbn.cQm/doc/openmap-archhtml#toc5 


MapObJects  -  Java  Edition  and  ArcGlS  (C/JMTK) 

Pros:  ArcGIS  is  a  fully  mature  industry  standard  mapping  kit.  It  has  every  feature 
we  may  want  and  more.  The  C/JMTK  version  comes  with  multiple  extensions 
including  a  Java  implementation. 

Cons:  Our  interface  to  ArcGIS  would  be  be  MapObjects-  Java  Edition,  which  is 
little  more  than  a  basic  mapping  tool,  probably  not  as  powerful  as  OpenMap.  All 
ArcGIS  created  data  would  have  to  be  served  to  MapObject  layers.  If  we  can't  get  a 
government  sponsor  for  the  C/JMTK  version  of  this,  it  is  most  likely  out  of  our  phce- 
range. 

Licensing:  Possibly  free  under  OEAVT. 

Notes: 

•  Capability  matrix:  htt  p  ://www .  e  sri .  co  m  /I  ibra  ry  /  bro  ch  u  re  s/p  dfs/a  rep  i  s9  2- 
fu  nctiona  litv-matrix.  Pdf 

•  MapObjects-Java  Edition: 

http://www.esn  .com/softwa  re/moia  va/a  bo  ut/o  verview.html 
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OSS/M 

Pros:  OSSIM  is  a  library  that  has  substantial  capability  to  load  all  the  file  types  we 
need  and  render  in  all  the  projection  we  want 

Cons:  OSSIM  is  not  a  Mapping  development  toolkit  and  really  has  no  built  in 
mapping  functionality  beyond  the  mention  file  loading  and  image  rendering.  All 
mapping  functionality  would  have  to  be  implemented  in  house.  In  addition^  many 
image  manipulation  capabilities  would  have  to  be  internally  developed  such  as  the 
geospatial  tiling  of  the  GIB  and  DTED  imagery 

Licensing:  Open  Source 

Notes: 

*  Website:  http : // www . o ssi m . o rq/ O SS I M AA/e loo m e . htm  I 


Google  Earth  (GE) 

Pros:  Google  Earth  is  a  fully  functional  3D  mapping  tool  that  out-of-the-box  supplies 
quite  a  lot  of  data— terrain  imagery  and  elevation,  geopolitical  data,  road  networks, 
and  more.  It  is  fast  and  seems  reliable.  User  supplied  data  is  loaded  from  a  format 
called  KML.  which  is  based  on  XML  and  can  be  easily  produced. 

Cons:  Google  Earth  is  first  and  foremost  a  standalone  viewer  of  its  own  data.  It  is 
not  capable  of  having  its  GUI  configured  nor  can  it  be  integrated  into  a  user 
application.  Although  it  does  handle  some  user  supplied  data,  specifically  basic 
raster  data  and  vector  datat  it  is  not  currently  capable  of  processing  more 
sophisticated  GIS  data  such  as  DTED  and  CADRG  Layering  handling  is  very  poor, 
verging  on  impossible.  Polygonal  data,  for  example  is  layered  arbitrarily  depending 
on  what  points  make  up  the  polygon. 

Licensing:  Basic  is  free.  Pro  has  a  yearly  fee  of  around  $400.  This  also  may  fall 
under  the  same  category  as  Google  Maps  in  which  the  free  licensing  only  applies  to 
publicly  available  apps.  There  is  no  dev  kit. 

Notes: 

•  Has  a  COM  API  that  includes  remote  control  of  the  camera,  loading  KML 
data,  and  retrieving  Feature  information,  (see 
http://earth.qQQqle.CQm/oQmapi/interfacelApplicationGE.html) 

•  GE  can  be  imbedded  in  a  Windows  App 

(see  http: //interactive earth  .bloqspot.com/2QQ7/01  /im  portinq-qis-data-i  nto- 

qooqle-earth.html) 

•  GE  has  an  enterprise  version  that  lets  you  serve  your  own  data  (raster,  GIS, 
etc.).  Word  on  the  Net  has  prices  ranging  from  $20K  -  $1 OOK.  Here  is  an 
overview  of  the  GE  Enterprise  capabilities: 
http://earth.qooqle.com/eai1h  enterprise.html. 
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Google  Maps  (GM) 

Pros:  None. 

Cons:  The  GM  API  has  no  handling  of  raster  imagery.  GM  is  only  available  as  a 
thin  client. 

Licensing:  Free  for  public  use;  private  use  has  associated  fees  starting  at  $10K. 
Notes: 

*  The  GM  API  is  javascript. 

*  The  API  is  documented  here; 
http://www.qooqle.com/apis/maps/documentatjon/ 

*  Google  Maps  can  be  made  to  show  raster  data  with  SVG  plug-ins,  but 
appears  to  run  slow,  plus  all  layer  handling  would  have  to  be  implemented  in- 
house.  f  http://www.web-maps.com/GooqleMapExplQre/) 

■  Here  is  another  discussion  about  overlaying  raster  imagery  onto  a  google 
map.  It  involves  using  MapServer  fhttp://m a pserver.ais.umn.edu/)  among 
other  tools:  http://qroups.qaQqle.com/qrQup/GQoqle-Maps- 
AP I/browse  thread/thread/1717c535f611a207/838ac2b2a8db4b06?a=wms+n 

elson&rnum=2#838ac2b2a8db4bQ6 


JView 

Pros:  JView  is  highly  optimized  and  is  implemented  in  Java. 

Cons:  JView  is  meant  as  a  3D  Tenderer  of  imagery  onto  a  spheroid,  much  like 
Google  Earth.  It  currently  can’t  easily  handle  any  types  of  2d  data,  such  as 
Polygons  and  geographic  point  icons. 

Licensing:  Free  to  use  on  OEAVT  as  GOTS. 

Notes: 

*  Website:  h tt ps: //e xtra n et . rl . a f . m i I/ i v i e w/ 


uDig 

Licensing:  Open  Source  (LG PL) 

Notes: 

•  uDig  is  not  really  a  mapping  development  kit,  but  an  open-source  application 
developed  on  the  Eciip^  Modeling  Framework.  Although  it  is  extendable,  it  is 
not  what  we're  looking  for 
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Recommendation  &  Discussion 


Development  of  the  GEV  based  on  OpenMap  is  recommended,  based  on  the 
availability  of  key  functions  ,  its  existing  use  with  the  AOC,  specifically  within  IWPC, 
and  with  SAIC’s  development  team’s  familiarity  with  the  tool.  SAIC  intends  to 
proceed  with  initial  GEV  development  on  this  basis  absent  other  direction.  We  will 
be  performing  risk-mitigation  activities  (e.g.,  assessing  performance  with  hundreds 
of  layers,  extending  the  lambert  projection  to  the  southern  hemisphere,  etc.).  Some 
raster-heavy  functions  needed  by  the  GEV  may  be  achieved  using  external 
packages  to  pre-process  data  into  more  manageable  formats. 

Why  not  Google  Earth?  The  ability  to  depict  2D  maps  in  a  low- distortion  conformal 
projection  is  judged  to  be  an  essential  element  for  accurately  understanding  the 
disposition  of  forces  and  events  and  rules  out  Google  Earth,  or  virtually  any 
perspective  or  orthographic  only  display  for  the  GEV.  The  figures  below  illustrate 
the  issue. 


Figure  1:  Small  scale  distortion  comparison:  Lambert  (black),  Azimuthal- 
Equidistant  (red),  and  Perspective  (green). 

For  depicting  small  areas,  nearly  any  common  projection  is  acceptable,  as  shown  in 
figure  1 .  Correlation  is  noted  best  at  the  center  of  the  projection  (Iraq),  with 
increasing  distortion  with  distance.  Note  that  the  perspective  projection  (Google 
Earth)  is  under-representing  the  distance  at  the  edge  of  the  map.  The  azimuthal- 
equidistant  has  the  property  of  maintaining  true  scale  on  all  meridians  and  on  the 
standard  parallels.  The  Lambert  projection  approximates  this,  with  the  added 
property  that  great-circle  routes  are  approximately  straight  lines  (thus  typical  ground 
points  underlying  great  circle  air  navigation  routes  are  correctly  depicted).  Use  of 
NGA  standard  parallels  and  the  Lambert  projection  will  also  result  in  depictions 
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essentially  identical  to  paper  GNC,  JNC  JOG  andTPC  maps  and  charts  familiar  to 
USAF  operators. 


Figure  1:  Large  scale  distortion  comparison:  Lambert  (black),  Azimuthal- 
Equidistant  (red),  and  Perspective  (green). 

Predictably,  a  larger  scale  map  exhibits  greater  distortion.  This  view  is  centered  on 
Somalia.  Note  that  at  the  extremes  of  the  map,  the  perspective  projecting  under¬ 
depicts  distances  by  roughly  15%  of  the  map  extent.  While  useful  in  3D  rendering, 
the  perspective  projection  is  not  conformal  and  preserves  neither  true  azimuth,  area 
or  distance  relationships. 

Also  of  concern  on  both  Google  Earth  and  Google  Maps  are  licensing  terms, 
especially  for  applications  not  publicly  published. 

Why  notC/JMTK;  While  the  ESRI  ArcGIS  based  C/JMTK  is  clearly  the  most 
capable  GIS  software  considered,  it  carries  significant  development  risks  for  the 
GEV: 

-  The  GEV  GUI  is  intended  to  be  highly  optimized  for  the  assessment  task, 
mixing  elements  of  layer  control/order  with  plan  hierarchy.  Achieving  this  is  C/JMTK 
appears  to  be  complex,  requiring  significant  coding  from  scratch. 

-  The  complexity  and  lack  of  local  experience  with  the  ArcGIS  API  increases 
risk. 

-  The  available  Java  implementation  within  the  C/JMTK  does  not  appear  to 
significantly  differ  from  OpenMap. 
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GEV  GUI  Design  Mockups 


Icon  Set 
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Here  are  some  examples  of  the  weighted  status  indicators  that  surround  the  icons  in  the 
GEV. 


Failing  Caution  Success 


d _  Wt:  0.0 

Op;  30% 


D-32 


GEV  displaying  a  TO 


The  plan  element  being  displayed  is  shown  in  the  upper  left  corner  in  it’s  GEV 
assessment  format. 


All  assessment  data  for  the  displayed  plan  element  is  shown  on  the  GEV.  There  is  a 
sidebar  to  the  right  for  non-geospatial  assessment  data,  as  well  as  out  of  view 
assessment  data. 


Assessments  with  an  Area  of  effect  have  an  unfilled  polygon  surrounding  them.  The 
TO's  polygon  is  a  bounding  polygon  created  from  aggregating  the  points  and  areas  of 
it’s  children. 
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GEV  displaying  a  TO  -  partial 


The  view  has  been  panned  upward  moving  part  of  the  assessment  off-screen. 

Out  of  view  assessment  is  displayed  along  with  non-geospatial  assessment,  in  the 
sidebar. 
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GEV  displaying  the  expanded  TT 


A  Tactical  Task  has  been  expanded.  Targets  are  shown  as  small  dots  in  the  same  color 
as  the  assessment. 
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GEV  displaying  the  collapsed  00 


The  parent  00  of  the  TO  has  been  collapsed.  The  TO  now  has  a  single  representative 
icon.  The  TO’s  assessment  data  has  been  simplified  to  single  smaller  representative 
circles.  There  are  two  other  assessment  areas  that  some  of  their  area  can  be  seen,  but 
not  their  icon,  so  they  are  placed  in  the  out  of  view  sidebar.  There  is  also  one  non- 
geospatial  assessment  data  in  the  sidebar. 
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APPENDIX  E 


GEM-S  10  ASSESSMENT  PROTOTYPE  VISUAL  SPECIFICATION 
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Detailed  Assessment  Graph 
(10  Assessment)  -  Plot  Details 


The  ellipses  indicates  confidence, 
its  major  and  minor  axis  adjust 
independently  to  reflect  confidence 
for  both  Direction  and  Magnitude. 
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diminishing  the  older  the  plots  are. 
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Here's  the  indicator  lists  and  sub  rollup  list  of 
this  ‘'blown  up"  plan  element  assessment 
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Element  Name 
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Rollup  Info  for  this  element  (see 
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Detailed  Assessment  Graph 
(Active  10  Assessment)  -  Plot  Details 


100% 


The  ellipses  indicates  confidence. 
Its  major  and  minor  axis  adjust 
independently  to  reflect  confidence 
for  both  Direction  and  Magnitude. 
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Blue  =  Desired  Effect;  Red  = 
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per  the  effect  data. 
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Indicator  Summary 
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